Copyright 1990 by the American Psychological Association, Inc. 0022-006X/90/S00.75

Journal of Consulting and Clinical Psychology 1990, Vol. 58, No. 6, 729-740

Empirical and Clinical Focus of Child and Adolescent Psychotherapy Research Debra Bass

Alan E. Kazdin

University of Pittsburgh School of Medicine

Yale University

Antoinette Rodgers

Wayne A. Ayers

University of Pittsburgh School of Medicine

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Yale University

The present study evaluated the characteristics of research on child and adolescent psychotherapy. Published studies (N = 223) of psychotherapy from 1970 to 1988 were codified to characterize research, clinical, and methodological characteristics. The major results indicate that (a) treatment research focuses almost exclusively on the impact of treatment techniques with scant attention to influences (child/adolescent, parent, family, therapist) that may moderate outcome and (b) several characteristics of the children/adolescents and methods of treatment delivery and approaches depart markedly from those evident in the practice of treatment. Priorities for treatment research to place clinical practice on firmer empirical footing are discussed.

Developing and identifying effective treatments of emotional and behavioral disorders of children and adolescents are high priorities for research. The range of dysfunctions, their relatively high prevalence rates, and direct costs of untreated psychological disorders among youth underscore the need for effective treatments (Institute of Medicine [IOM], 1989). For example, in the United States alone, between 12 and 17% (or approximately 7.5-14 million) of the nation's youth suffer from emotional and behavioral disorders (IOM, 1989; United States Congress, Office of Technology Assessment, 1986). Although psychotherapy cannot be viewed as the means to address all of the mental health problems of children and adolescents, it is also clear that effective treatments, once identified, would have widespread use and value. Recent reviews suggest that alternative forms of psychotherapy for children and adolescents are effective (e.g., Casey & Berman, 1985; Kazdin, 1990; Weisz, Weiss, Alicke, & Klotz, 1987). However, the conclusion has to be heavily qualified because of the restricted quantity, quality, and focus of psychotherapy research. The quantity of the evidence can be lamented on separate counts. At least 230 different forms of therapy are in use for children and adolescents, only a small fraction of which have been evaluated empirically (Kazdin, 1988). Among those treatments that have been investigated, relatively few studies are available. The dearth of studies is particularly evident in relaWe gratefully acknowledge support for the present project from The Robert Wood Johnson Foundation and from a Research Scientist Development Award (MH00353) from the National Institute of Mental Health. Also, we wish to express our gratitude to Mary Dulgeroff for her organizational acumen and management of the process leading to this report.

Correspondence concerning this article should be addressed to Alan E. Kazdin, Department of Psychology, Yale University, Post Office Box 11A Yale Station, New Haven, Connecticut 06520-7447. 729

tion to those treatments (e.g., individual psychotherapy, psychodynamically oriented therapy, family therapy, eclectic approaches) that are frequently used in clinical practice (Kazdin, Siegel, & Bass, 1990; Koocher & Pedulla, 1977; Silver & Silver, 1983). The underdeveloped child psychotherapy literature is particularly conspicuous when contrasted with research on the treatment of adults (e.g., Garfield & Bergin, 1986; Goldfried, Greenberg, & Marmar, 1990). The quality of psychotherapy research with children and adolescents has also attenuated the strength of the conclusions about treatment efficacy. The research has been criticized because of such methodological problems as the absence of suitable (e.g., randomly comprised) control groups; use of global outcome measures; failure to specify characteristics of the sample and their clinical dysfunction; lapses of or failure to monitor treatment integrity; small sample sizes; and little follow-up data, to mention a few (e.g., Barrett, Hampe, & Miller, 1978; Heinicke & Strassmann, 1975; Kazdin, 1988; Shaffer, 1984). These concerns do not of course apply to all studies. However, application of the concerns to the already small cadre of investigations limits further the strength of conclusions that can be drawn. Finally, the restricted focus reflects the departure of treatment research from the manifold conditions in which treatment is conducted in clinical practice. Most studies of child and adolescent psychotherapy utilize nonreferred cases recruited from the schools; the cases often evince mild and subclinical problems, and receive relatively brief treatment. These characteristics depart rather markedly from the cases and methods of treatment delivery in clinical settings (e.g., Kazdin et al, 1990; Koocher & Pedulla, 1977; Silver & Silver, 1983; Tuma & Pratt, 1982). The restricted focus of research is also reflected in the emphasis on alternative approaches as the salient, if not exclusive, source of influence. Scant attention is accorded other types of influences including child/adolescent, family, or therapist characteristics; treatment processes; or cir-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

730

KAZDIN, BASS, AYERS, AND RODGERS

cumstances of treatment administration that may moderate outcome. Lamentations of the paucity, caliber, and focus of research do not automatically point to needed directions for the field. More, better, and less restricted studies provide rather nonspecific directions. The research priorities depend in part on the goals of therapy research and the extent to which these goals are systematically approached. The goal of child and adolescent psychotherapy research is twofold. The initial goal is to understand alternative forms of treatment, the means and mechanisms through which they operate, and their impact on adaptive functioning. Research designed to understand psychotherapy is guided by the oftcited question, "What treatment, by whom, is most effective for this individual with that specific problem, under which set of circumstances?" (Paul, 1967, p. 111). Elaborations of the question convey that the research goal is to elucidate the effects of treatment, the conditions that influence outcome, and the processes by which change is achieved (e.g., Goldfried et al., 1990; Kiesler, 1971; Lambert, Shapiro, & Bergin, 1986). A related goal is to place clinical practice on firm empirical grounds. Although this latter goal is served by understanding treatments and how they operate, the extension, generality, and applicability of research findings to clinical practice cannot be assumed. Many treatments and conditions of treatment delivery utilized in clinical practice are rarely addressed in research. Research in child and adolescent psychotherapy can be examined by the extent to which it speaks to the manifold conditions (e.g., clinical problems, treatment approaches) of psychotherapy that characterize clinical practice. The purpose of the present study was to describe and characterize psychotherapy research for children and adolescents and to do so in a way that would permit evaluation in relation to clinical-research objectives of the field. The focus of research was examined to identify the types of questions that guide treatment studies and the extent to which the evidence is accumulating to understand the effects of treatment and the diverse conditions that may moderate therapy outcome. The oft-cited question, noted previously, draws attention to the range of variables that warrant scrutiny in relation to treatment outcome (Kiesler, 1971). The extent to which studies address diverse influences was examined to identify lacunae in the research on child and adolescent psychotherapy. The clinical focus of research was examined to evaluate the types of cases, clinical dysfunctions, treatments, and other features that characterize treatment research. The characteristics of clinical practice have been evaluated in several surveys of mental health practitioners involved in direct care of children and adolescents (Kazdin et al, 1990; Koocher & Pedulla, 1977; Silver & Silver, 1983; Tuma & Pratt, 1982). The extent to which research addresses or departs from conditions of clinical work has been less well studied. The primary objective was to examine the substantive focus of research in relation to the clinical-research goals noted earlier. The research and clinical foci were examined by evaluating published treatment research. A secondary objective was to describe methodological features of research. Apart from basic descriptions (e.g, types of groups, assessment foci, and methods), selected design features were examined. These features

(e.g., method of case assignment to conditions, evaluation of treatment integrity, assessment of follow-up) reflect dimensions and practices that may affect the quality of studies and the research base. Effect sizes were also examined to evaluate the impact of treatment in relation to comparisons commonly conducted in treatment research and to serve as a basis for design decisions (e.g, sample size) in therapy trials. In brief, the present study describes characteristics of child and adolescent psychotherapy outcome research. The descriptive focus was intended to permit an evaluation of the extent to which research addresses the range of factors that can influence outcome, the conditions that characterize clinical practice, and methodological features that may affect quality of inferences that can be drawn about treatment. The purpose was to codify characteristics of research and to use the information as a basis for identifying research priorities.

Method Selection of Psychotherapy Studies Definition of psychotherapy. We examined published outcome studies of psychotherapy for children and adolescents. Psychotherapy was denned as an intervention designed to decrease distress, symptoms, and maladaptive behavior and/or to improve adaptive and prosocial functioning. The type of intervention included some form of counseling, structured or unstructured interaction, training program, or plan, or drew upon psychosocial influences such as discussion, learning, persuasion that focuses on how clients feel (affect), think (cognition), or act (behavior) (Garfield, 1980; Waldron-Skinner, 1986). Outcome investigations referred to studies designed to measure psychological adjustment or functioning after treatment was completed (posttreatment). Excluded from the definition were interventions using medication as a form of treatment or interventions directed singularly at educational, career, or vocational goals. Literature search. We conducted a computer search to identi fy psychotherapy research completed with children and adolescents. The search focused on studies completed from 1970 to the present (i.e., mid-1989 when the search was completed). General search words (e.g., psychotherapy, counseling, treatment) and terms for specific treatment techniques were used. The words were obtained from other studies of psychotherapy in which searches were completed (e.g., Casey & Berman, 1985; Weisz et al, 1987) and from treatments included in surveys of clinical practice (e.g, Koocher & Pedulla, 1977; Silver & Silver, 1983; Tuma & Pratt, 1982).' The search yielded 3,086 abstracts of articles on child and adolescent psychotherapy. Criteria. From the abstracts, articles were identified for in-depth evaluation based on several criteria. Studies were included if (a) the article was an empirical investigation of treatment, rather than a review, discussion, or case study; (b) persons in the study included children or adolescents, as defined by ages 4-18 years; (c) at least two groups were included in the study, one of which received a form of psychotherapy as defined previously, and the other of which included some other condition such as no-treatment or waiting-list controls, or an alternative treatment; and (d) measures were included to evaluate treatment outcome and focused on adjustment, psychological functioning, or other features designed to reflect psychological well-being. Studies were excluded if (a) the intervention group consisted of medi-

1 A full list of search words for identifying studies, a list of the final set of studies, and the coding form can be obtained from Alan E. Kazdin.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

SPECIAL SERIES: CHILD AND ADOLESCENT PSYCHOTHERAPY cal treatment (e.g., medication), educational experiences directed toward academic knowledge or general classroom management, career counseling or decision making directed toward occupational or academic goals, and rehabilitation, occupational, or physical therapies directed toward motor skill performance; (b) the intervention consisted of a hospital, institutional, residential, or day-treatment program; and (c) the study focused on prevention in which no clear problem or maladaptive behavior was yet evident (i.e., primary prevention). To invoke the selection criteria, two master's degree level research associates independently reviewed each abstract and indicated whether the criteria were met. Abstracts for which there was consensus for inclusion or exclusion were placed in their respective categories. Cases of doubt or discrepancy between the two raters were evaluated by a third rater and then discussed to reach consensus. If any doubt remained, the full article was obtained for subsequent evaluation with the idea that inclusion and exclusion criteria could be better invoked from reading the full article. Set of studies for evaluation. From the original set (N = 3,086), 393 studies were selected for evaluation. The articles were obtained from libraries and photocopied. If the journal was unavailable within the university, the article was obtained through interlibrary loan, or a reprint request letter was sent to the author of the study. Of the 393 articles, 166 were excluded after the full article was read and selection criteria invoked; 8 additional articles could not be obtained after repeated library searches and at least two unanswered requests to authors for the reprint; 1 article was excluded because it was a duplicate of another published article that was included. We evaluated the remaining 218 articles. These articles spanned a 19-year evaluation period (1970-1988) and encompassed 87 different journals. Because 5 articles included more than one investigation, a total of 223 empirical investigations of child and adolescent psychotherapy met the inclusion criteria. These 223 investigations served as the basis for the present evaluation of the characteristics of child and adolescent psychotherapy research.2

Evaluation of the Studies Domains of interest. The purpose of the study was to characterize psychotherapy outcome research and to examine clinical, research, and methodological features. Three broad domains of interest were used to evaluate individual studies including (a) the substantive research questions that guided the study as reflected in the types of groups and group comparisons included to evaluate treatment; (b) characteristics of the patients, clinical problems, therapists, and treatments; (c) selected methodological features related to the evaluation of treatment outcome such as the method of assigning cases to conditions, assessment domains and methods, criteria for evaluating change, and evaluation of follow-up. Within each domain, several specific questions were addressed. As noted earlier, the oft-cited question that guides therapy research focuses on treatment and the diverse patient, therapist, and other conditions that may have an impact on outcome. To operationalize different facets of the question, the type of comparison, alternative groups, and treatment evaluation strategy were examined for each study. Within the clinical domain, studies were examined to identify the extent to which features of child and adolescent psychotherapy research encompassed cases, problems, and treatments that characterize clinical practice. Items were drawn from surveys, cited earlier, that have characterized the practice of child and adolescent psychotherapy. Several methodological characteristics of research were also examined to document such features as the method of subject assignment to conditions, number of subjects, groups and types of comparison groups, and measurement strategies. Effect sizes were computed to note the magnitude of differences for major comparisons that often

731

guide research (e.g., comparison of treatment with control conditions and comparison of alternative treatments with each other). The purpose was to provide information for use in the design of studies in relation to sample size and power. Development of the codes. The full text of each study (/V = 223) was read and coded in relation to research, clinical, and methodological domains, noted previously. All characteristics of interest from each article were scored on a 13-page coding form. The form included 55 major items. With subitems and separate queries, 202 different entries were scored for each article. Response formats varied among the items. Most of the codes required the rater to mark one of three alternatives indicating that the characteristic of the study was present, not present, or could not be discerned from the article. Occasionally, fill-in responses were required (e.g., number of therapists in the study, duration of treatment).

Procedures Four persons (two women, two men; three with master's degrees, one with a doctoral degree) served as coders. Three persons were mental health professionals involved directly in child and family-based psychotherapy and clinical research. The fourth coder was a biostatistician (master's degree) involved directly in child psychotherapy research. Each coder read each investigation and coded alternative characteristics. Before beginning the formal evaluation of the studies, coders practiced on a subset of studies. For training purposes, each person coded all features of a study. Each study was individually discussed in group meetings to identify agreements and disagreements. The discussions served as a basis to sharpen definitions of the response codes, to divide items that included embedded questions, or to eliminate redundant items. Discussion meetings were continued through several studies until further iterations of the definitions did not emerge. At that point, formal coding began. Articles were circulated to each coder. The coding form was divided into foursections merely for purposes of rating. Each person coded one of the sections for all of the studies. The section coded by a given rater remained constant for all of the articles, so that each coder would develop skills in identifying highly specific features of the studies. Evaluations were made independently. Meetings were held weekly to discuss general problems, ambiguities in the codes, or novel characteristics of specific studies. When all studies were coded, each coding form and article was rechecked by one coder who reread the original study. All instances where an item was scored as "cannot tell" were checked to identify whether information might be evident. Also, any coded response on the form with a question mark (which coders were instructed to use) or any ambiguity (erasures) on the coding form were checked.

2 Inclusion of 223 studies in the present study is difficult to evaluate out of context. The number compares favorably with other efforts to characterize the literature. For example, in the landmark meta-analysis of Smith, Glass, and Miller (1980), approximately 90 studies were included that focused on children and adolescent psychotherapy. Almost one half of these were unpublished dissertations and theses. In the two major meta-analyses of child and adolescent psychotherapy, 7 5 and 108 studies were identified (Casey & Berman, 1985; Weisz et al., 1987, respectively), 24 of which overlapped. Understandably, meta-analytic reviews have identified fewer studies than the present investigation because of the requirement to include studies in which effect size information could be obtained.

732

KAZDIN, BASS, AYERS, AND RODGERS

Results The purpose of the study was to characterize the research questions, clinical features, and selected methodological features. Clinical characteristics of the studies are presented first to describe the general features of research with children and adolescents.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Clinical Characteristics Cases and clinical problems. Several items were included to codify characteristics of children, their dysfunctions, and treatment. The goal was to evaluate the extent to which these characteristics resemble those of clinical practice. Table 1 provides characteristics of cases included in research studies of child and adolescent psychotherapy on the basis of the 223 investigations. Entries consisted of the percentage of studies that included various characteristics. The mean age of all cases treated in the research studies was 10.2 years (SD =3.3); the majority of cases fell within the ages of 6-11 years. Boys tended to be included more frequently than girls and constituted 67.3% of the cases. Although we examined racial/ethnic status of the cases, 79.8% (« = 178) of the studies did not specify this information. Of the available studies (n = 45) that did, White, Black, and Hispanic youth were included in 75.6%, 65.9%, and 17.9% of the studies, respectively. As noted in Table 1, other ethnic/racial groups were rarely included. Method of case recruitment was also examined. In the majority of studies (76.6%), cases were obtained through direct solicitation of volunteers. Relatively few studies utilized children drawn from outpatient clinic referrals (28.4%) or inpatient cases (3.3%). The type of problem that served as the focus of treatment was also evaluated. Different levels of coding the problems were adopted to capture the heterogeneity of the descriptions in the articles and to convey varying levels of specificity, as noted in Table 1. In terms of general descriptors, acting-out behaviors (50.7%) and behavior problems at home (22.0%) encompassed " the largest percentages of the studies. Other terms were used to classify problems more broadly in terms of internalizing (inward directed), externalizing (outward directed), and learning/ academic problems. As noted in Table 1, externalizing problems (e.g., overactivity, conduct problems) served as the most common focus in treatment research (47.3% of the studies). Child problems were also coded to approximate categories of the Diagnostic and Statistical Manual of Mental Disorders (3rd ed., rev.; DSM-III-R; American Psychiatric Association, 1987) including disorders arising in infancy, childhood, and adolescence plus other disorders (e.g., depression) not age-specific. The purpose was to characterize the general focus of outcome research in contemporary diagnostic terms. Diagnostic information presented in the study was used to classify problems. However, relatively few (14.3%) of the studies included a formalized diagnostic system. This may be due in part to the recent appearance of DSM-HI and DSM-III-R (1980,1987), which accord major attention to disorders of infancy, childhood, and adolescence. For present purposes, clinical problems were classified as approximations of diagnosable disorders within DSM-III-R.

Table 1 Characteristics of Children/Adolescents Seen in 223 Treatment Studies Characteristic Age group' 0-5 years 6-11 years 12-17 years 18 years or older Sex"

All males All females Race White Black Hispanic Asian American Indian How cases were obtained Solicited by investigator Clinic referred Inpatients Court referred Incarcerated Child problems: general descriptors Behavior problems at home Behavior problems at school Acting-out behaviors Emotional problems Learning problems Child problems: broad band Internalizing Externalizing Both internalizing and externalizing Learning/academic Other Disorders' (approximations) most frequently reported Conduct/oppositional Attention deficit/hyperactivity Anxiety Academic skills Enuresis/encopresis Mental retardation Depression/mood

% of studies 26.9 75.1 55.7 6.4 19.0 4.0

75.6 65.9 17.9 5.1 0.0 76.6

28.4 3.3 1.4

0.9

22.0 9.9 50.7 18.9 19.8 16.4 47.3 3.2 15.0 24.9 37.1 17.2 10.9 6.8 5.9 2.3 1.8

Note. Entries include the percentage of studies in which a characteristic was included. Within a given category (e.g., race, child problems), entries are not mutually exclusive. A given study could be scored for one or more alternatives within a category if separate characteristics were included. • Mean age of patients: 10.2 years (SD = 3.3). " Mean percentage of boys: 67.3 (SD = 25.1). c According to criteria listed in the Diagnostic and Statistical Manual of Mental Disorders (3rd ed., rev; DSM-III-R).

Table 1 presents disorders that were addressed in the largest percentage of studies. The most frequent foci of treatment research were conduct/oppositional disorder (37.1% of the studies), attention deficit disorder (17.2%), and anxiety disorder (10.9%). Other disorders not listed in the table (including eating disorders, adjustment disorders, parent-child problems, substance abuse, stereotyped habit, tic, psychotic, and organic disorders) were included in of studies

37.0 56.8 67.2 27-60 8-553 36.0 55.3 65.9 26-59 8-553 29.0 44.2 64.7 20-40 6-553 3.0 2-3 2.0 1.0

93.2 59.2 35.9 18.4 37.7 0.4 84.8 15.7 18.4 79.8 24.7 22.9 21.5 14.3 11.7 2.2

Characteristic Assessment modality Direct observation Paper and pencil Interview Psychophysiological Projective Other Source of information Observers/i ndependent raters Children/adolescents Parents Teachers Records (institutional, class) Peers Expert judges Therapists Other Assess clinical significance Type of clinic significance measure Normative comparison Magnitude of change Other Method of case assignment to groups Random Nonrandom Intact groups Other Treatment integrity Use of a manual/films Monitor/or assessment Follow-up (FU) Assess FU Mdn of longest duration (months) M

SD 25th-75th percentile (months) Range (months)

No.

% of studies 72.6 71.3

12.6 3.6 2.7 2.2 46.6 42.6 28.3 27.8 15.7

4.9 4.5 2.7 4.9 12.6 6.3 5.4 0.9

81.3 4.1 9.8 4.3 55.6 19.3 41.3 5.0 7.2 7.9

2-12 0.8-60

10.8

9.9 5.4

Note. Entries include the percentage of studies in which a characteristic was included. A given study could be scored for more than one alternative within a category.

Outcome assessment on alternative measures refer to specific assessment devices. Use of these or other indices to evaluate the clinical significance of therapeutic change was also assessed. Clinical significance refers to evaluation of the extent to which treatment produces an important or marked change in dysfunction. Efforts to assess the clinical significance of change were evident in 12.6% of the studies. As noted in Table 4, the two methods used to assess clinical significance were comparing treated individuals after treatment with a nonreferred (normative) comparison group and defining a certain magnitude of change (e.g., elimination of the presenting problem, degree of individual patient change). Evident from these data is the pau-

city of studies in which the clinical impact of treatment was evaluated. Drawing inferences about treatment. A few salient methodological features focused more specifically on the quality and type of inferences that might be drawn. Method of case assignment to groups is obviously pivotal for addressing several threats to experimental (i.e., internal) validity. As evident in Table 4, in 81.3% of the studies, cases were assigned randomly to groups. The fact that close to 20% of the studies did not use random assignment places potential constraints on or increases the complexity of interpreting a major segment of research. Treatment integrity, or the fidelity with which treatment was

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

736

KAZDIN, BASS, AYERS, AND RODGERS

relative merit and impact of individual treatment techniques. implemented, was coded. As noted in the table, 55.6% of the Consequently in computing ESs for treatment-treatment comstudies included written treatment manuals/or materials for parisons, the absolute value of the ES (|m, - m2|/s) was calcutherapists or filmed materials (in modeling studies) to standardlated. Thus, the mean ES for a study with two treatments would ize treatment presentation to the children/adolescents. If any be the mean of the absolute value differences between each of attempt was reported to monitor treatment implementation, to these treatments for each of the outcome measures. provide ongoing supervision of therapists during treatment, or Table 5 presents estimated ESs for the comparisons of interto evaluate the execution of treatment during its implementaest and the number of studies of 223 for which estimates could tion, the study was considered as monitoring treatment integbe calculated. As a potential guideline for interpretation of the rity; 19.3% of the studies monitored or evaluated treatment indata, it is useful to bear in mind Cohen's (1988) classification of tegrity. small, medium, and large ESs as .20, .50, and .80. When treatThe assessment of follow-up was also examined, given that ment is compared with no treatment, the median ESs at postinferences drawn about the efficacy of treatment may change treatment and follow-up were .72 and .88, respectively. These depending at the point in time (posttreatment, follow-up) that ESs reflect directional change (improvement) of treatment vertreatment is evaluated (see Kazdin, 1988). In the present samsus no treatment and fall within the range of large ESs. When ple, 41.3% of the studies included follow-up assessment. If foltreatment is compared with active treatment control condilow-up was assessed on multiple occasions within a study, only tions, the median ESs at posttreatment and at follow-up were the follow-up of the longest duration was coded. The median .57 and .31, respectively. These ESs indicate that treatment is duration of the longest follow-up was 5 months. Using the 25th generally better than active control conditions. The ESs are in and 75th percentiles, 50% of the studies with follow-up assessed the medium to small range. When two or more treatments were cases between 2 and 12 months after treatment ended. compared with each other, the median absolute ESs at postEffect size (ES) was examined to evaluate the magnitude of treatment and follow-up were .47 and .56, respectively. The outcomes evident among salient group comparisons. The purmagnitude of these fall within the range of medium ESs. pose was to provide a basis for planning future studies, given Effect size is of interest in relation to the design of research. the critical methodological decisions (sample size, power) reThe ability to detect differences between conditions when they lated to treatment evaluation. Effect sizes were estimated for (a) exist (i.e., power) is a function of alpha, sample size, and ES. comparison of treatments) with a no-treatment (or waiting-list) Sample sizes are relatively small in psychotherapy research studcontrol condition, (b) comparison of treatments) with an active ied in the present paper. As noted previously, at posttreatment (e.g., attention placebo) control condition, and (c) comparison of and follow-up, median sample sizes were 36 and 29, respeceach treatment with other treatments included in the study. tively. The median number of groups was 3. If sample size (N) is Within each study, an ES was calculated between each pair of divided by the median number of groups, group size (ri) at postgroups on each outcome measure.3 Each ES was classified as treatment and follow-up was approximately 12 and 10, respeccoming from a comparison of treatment versus no treatment (T tively. Evaluations of statistical power in clinical research invs. NT), treatment versus active control (T vs. AC), or between cluding psychotherapy studies suggest that sample sizes of this two treatments (T, vs. T2). Also, each ES was computed for magnitude are likely to detect differences only with large ESs posttreatment and follow-up. If multiple follow-up assessments were reported in a given study, only the last (longest duration) assessment was considered. Thus, there were 6 ESs that might 3 derive from a single study. Within a study, multiple values of Effect size (ES) was defined as (mi - m2)/s, where m, and m 2 refer each ES might be obtained if there were several outcome meato two group means (e.g., treatment or control) and s is the pooled within-group standard deviation. In computing ESs, the pooled estisures. The ESs of each type within a study were averaged, so mate was used in the present analyses because it is readily estimated that each study contributed no more than one mean ES per from studies where t and Ftests are reported but where standard deviacomparison of interest.4 Effect sizes were calculated from tions for individual groups are omitted. In addition, the pooled estimeans and standard deviations reported in the studies. When mate may be a less biased estimate than the standard deviation of the this information was unavailable, ESs were calculated from control group (Hedges & Olkin, 1985). other reported statistics, as described elsewhere (Smith et al., 4 Within a given study more than one outcome measure was likely to 1980). be included. In some previous meta-analyses, separate outcome meaEffect sizes were calculated between treatment and no treatsures have been used as separate ESs for the data analyses. The issues ment and between treatment and active control groups to idenraised with this procedure include the undue weight given to studies tify the magnitude of improvement or decrement that treatwith a large number of outcome measures and the nonindependence of observations (ESs) for any data analyses. An alternative strategy is to ment provided. Thus, ESs could assume negative and positive calculate a mean ES for a given comparison (e.g., T vs. NT) by averagvalues in an individual study. Effect sizes for instances in which ing the ESs from the individual outcome measures. Thus, for a given two or more treatments were compared were treated differently. study with a treatment and no-treatment control condition, one ES The interest of the present evaluation was in the magnitude of would be generated for that comparison on the basis of the mean of ES when this type of comparison (T, vs. T2) was made without ESs for all of the outcome measures. Alternative strategies for handling regard to drawing conclusions about specific approaches to multiple outcomes within a study have different limitations. A major treatment. Thus, in the present study, the specific conditions limitation of the present method is that it assumes that all outcome that constituted T, or T2 (e.g., psychodynamic therapy, cognitive measures in a study should be. weighted equally (i.e., are equally importherapy) were not of interest. This stands in contrast to metatant). There remains no consensus on how to prioritize outcome meaanalyses where ESs between treatments are used to evaluate the sures to resolve this concern (see Brown, 1987).

SPECIAL SERIES: CHILD AND ADOLESCENT PSYCHOTHERAPY Table 5 Estimated Effect Sizes (ESs) for Different Comparisons Comparison Treatment vs. no treatment Mean ES SD

MdnES 25th-75th percentile Range No. of studies Treatment vs. active control Mean ES This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

SD

MdnES 25th-75th percentile

Range No. of studies Treatment vs. treatment* Mean ES SD

MdnES 25th-75th percentile Range No. of articles

Posttreatment

Follow-Up

0.88 0.74 0.72 0.38-1.28 -0.02-4.08 64

0.89 .68 0.88 0.24-1.15 0.00-1.92 10

0.77 0.71 0.57 0.25-1.01 -0.03-3.53 41

0.39 0.36 0.31 0.23-0.44 -0.02-1.52 15

0.59 0.56 0.47 0.28-0.71 0.00-3.99 70

0.64 0.73 0.56 0.26-0.72 0.00-4.15 38

* Effect size for this comparison is based on absolute differences between treatments (i.e., without reference to the direction of these differences).

(see Kazdin & Bass, 1989; Rossi, 1990). These are the ESs within the range of comparisons of treatment versus no treatment. Yet much of the research compares alternative treatments or variations of a given treatment. The small to medium ESs evident in this type of comparison are less likely to be detected.

Discussion The present study evaluated published research on child and adolescent psychotherapy. The evaluation focused on (a) the extent to which research addresses the effectiveness of alternative treatments and the conditions that may influence treatment outcome and (b) the extent to which research encompasses cases, treatments, and other features that characterize clinical practice. The Focus of Psychotherapy Research The goal of research, as noted earlier, is to understand psychotherapy and the mechanisms and conditions that moderate its effects. A wide range of conditions such as the type of patient, therapist, treatment, and so on can influence treatment outcome. The present findings indicate that the majority of research focuses on evaluating some facet of the treatment approach or technique. Typically, studies compare treatment to a no-treatment or active treatment control group or contrast alternative treatment groups. This was evident in the specific group comparisons as well as the strategies (e.g., package, comparative, dismantling, parametric) that characterize research. Scant attention is accorded evaluation of nontechnique variables (e.g., classification variables to divide the groups) that may

737

moderate treatment outcome. The oft-cited research question draws attention to a host of conditions other than treatment technique that might influence change. Researchers have consistently encouraged such a focus to avoid the implicit view that a given treatment is likely to operate in a uniform fashion across all conditions (e.g., Kiesler, 1971; Lambert et al., 1986). However, the published research generally focuses heavily on treatment technique alone. Clinical Characteristics of Treatment Research In addition to examining the focus of research, we also evaluated characteristics of treatment research in relation to clinical practice. The characteristics of clinical practice have been charted in various surveys of practitioners (Kazdin et al., 1990; Koocher & Pedulla, 1977; Silver & Silver, 1983; Tuma & Pratt] 1982). We examined clients, their dysfunctions, features of treatment delivery, and treatment approaches that are studied in psychotherapy research. Several characteristics of research evident in the present study parallel characteristics of clinical practice. The tendency to focus on youth within the 6- to 11year age group, the slightly greater attention to boys than girls, and attention to dysfunctions that approximate conduct/oppositional disorder, attention-deficit/hyperactivity, and anxiety disorders are common foci in research sampled in the present study and in clinical practice. Although a few characteristics are similar across research and practice, the discrepancies are more salient. Our findings indicate that in treatment research, the bulk of the cases are children solicited from and treated in the schools. The most common form of treatment delivery is in groups. In contrast, clinical practice usually focuses on clinically referred youth who are seen in clinics, mental health agencies, or private practice. Treatment is usually provided individually. The major discrepancy is not of course the different settings (schools vs. clinics). It is likely that the types of cases seen in clinics may be somewhat different in the severity and dysfunctions or problems they present. An example of differences in the type of dysfunction is illustrated by the treatment of adjustment disorder. In the present evaluation, no studies were found in which adjustment disorder, or dysfunction approximating that, was treated. In contrast, adjustment disorder is frequently the basis for treatment of children in clinical work (Kazdin et al., 1990; Silver & Silver, 1983; Tuma & Pratt, 1982). Major discrepancies between treatment as studied in research and as conducted in clinical practice are evident in treatment delivery. Our findings indicate that the bulk of treatment research focuses on behavior modification and cognitive-behavioral techniques. These are often used in clinical practice. However, in practice greater attention is accorded to individual psychotherapy, psychodynamically oriented psychotherapy, family therapy, and eclectic treatment than is evident in psychotherapy research. The duration of treatment too varies markedly across clinical work and research. The present study and other evaluations of research have indicated that treatment is relatively brief (with means from separate evaluations that span 8-10 weeks; Casey & Berman, 1985; Weisz et al., 1987). In clinical practice, treatment is much longer (with means from separate evaluations that

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

738

KAZDIN, BASS, AYERS, AND RODGERS

span 27-55 weeks; Kazdin et al., 1990; Silver & Silver, 1983). These differences remain evident even when the evaluation is restricted to the same dysfunctions (e.g., conduct disorder, attention deficit disorder, and anxiety disorder) seen in both research and practice (see Silver & Silver, 1983). Other features of treatment delivery such as the involvement of the parents, seeing the family as a unit, and involvement of and consultation with teachers as part of treatment are more common in clinical practice than in treatment research. In general, several conditions that characterize clinical research depart from the conditions of clinical practice. The discrepancies do not necessarily imply that the results from the restricted conditions of research will not apply to the conditions of clinical practice. At the same time, the discrepancies alert us to the prospect that manifold conditions of clinical practice and their role in therapy process and outcome remain to be examined in research. Methodological Characteristics

The likelihood exists that power of studies with these sample and effect sizes will be lower than recommended for research (see Kazdin & Bass, 1989; Rossi, 1990). Restrictions of the Present Evaluation Several limitations can be identified in the present study. The first pertains to the selection of studies on which conclusions were based. Several criteria were imposed in the search for journal articles. The selection criteria may have yielded a narrow set of studies and may foster conclusions with commensurately limited generality. In support of this concern, some studies such as those detailed in books (e.g., Feldman, Caplinger, & Wodarski, 1983; Kolvin et al., 1981) were not included in this study. Whether a further sampling from different sources (e.g., books, unpublished studies) would generate different conclusions or research priorities is a matter of surmise. Second, the coding of studies and evaluation of alternative dimensions may reflect preconceptions on the part of the present investigators and point in the direction of, if not dictate, some of the conclusions. The codes for evaluating individual investigations were developed for the present study and hence are heir to this criticism. Errors of commission in the codes might be engendered by our views of the critical categories for evaluation (e.g., research questions, treatment evaluation strategies). Omissions might be expected from our prioritization of questions and oversights. At the same time, within the limits of our codes, the findings point to several areas in need of research.

Selected methodological characteristics of research were also studied to catalogue current practices and to identify specific features that may affect the strength of inferences. A few salient findings can be noted that may well affect the conclusions drawn from psychotherapy research. First, random assignment of cases to conditions was evident in approximately 80% of the studies. The relatively large segment of studies without random assignment may reflect constraints placed on treatment research or possibly specific constraints of school settings where the bulk of treatment research was conducted. In any case, a significant segment of research begins with conditions that can Priorities for Psychotherapy Research influence the quality of inferences or at least make evaluation The present findings address a circumscribed set of research, difficult. clinical, and methodological features of child and adolescent Second, results indicate that provisions for treatment integpsychotherapy research. Nevertheless, from these a number of rity are evident in slightly more than one half of the studies. priority areas might be identified to guide research. The bulk of However, only about one fifth of the studies report checking, psychotherapy research focuses on the effectiveness of alternamonitoring, or assessing the extent to which treatment is carried out as intended. tive treatments and variations of a given technique. This focus Whether studies assess the status of cases at follow-up was cannot be faulted in its own right because the vast majority of also examined. Follow-up was assessed in approximately 40% available treatments for children and adolescents have not been of the studies. The number represents a lower bound estimate evaluated empirically (Kazdin, 1988). At the same time, many because investigators occasionally report follow-up data sepaof the techniques frequently used in clinical work (e.g., psychorately from posttreatment results. The attention to follow-up is dynamically oriented therapy, family therapy, play therapy) sufcritical because conclusions about treatment outcome can vary fer from empirical neglect (Kovacs & Paulauskas, 1986; Tuma widely from posttreatment to follow-up (Kazdin, 1988). & Sobotka, 1983). These and other techniques shown to be in Effect sizes were also examined for comparisons commonly widespread use represent special research priorities. Combinaincluded in research. Effect sizes have been used in metations of different treatments also warrant attention. Practianalyses to evaluate the effects of specific treatments (e.g., Casey tioners report the widespread use and utility of eclectic ap& Herman, 1985; Weisz et al, 1987). In regard to the impact of proaches in their practice with children and adolescents (Kaztreatment, the present results are consistent with the other redin et al, 1990). The manner in which different techniques are views in the magnitude of effects accorded therapy in relation deployed and combined warrant further specification so they to no treatment (median ES at post = .72, at follow-up = .88). can be evaluated in research. Effect size was included in the present study to address the To move beyond technique variables, greater effort is needed methodological and design issue. For many comparisons of into study patient (child/adolescent), parent, and family variables terest (e.g., comparing different treatments or variations of in relation to treatment outcome. Investigations of the factors treatment), small to medium ESs are likely (Cohen, 1988). The that may moderate outcome might be based on theory underlytypical sample size for studies included in the present evaluaing individual treatments and the factors that moderate outtion (e.g., 37 cases at pretreatment, divided among three groups) come. Little therapy research appears to be theory-driven, alis likely to provide a weak test of the comparisons of interest. though many approaches specify critical variables and pro-

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

SPECIAL SERIES: CHILD AND ADOLESCENT PSYCHOTHERAPY

cesses (e.g., cognitive events, facets of family interaction, treatment processes) on which outcome is considered to hinge. Efforts to examine such influences in relation to outcome might be more systematically extracted from theoretical tenets underlying treatment. Clinical expertise and experience may also serve as a basis for identifying promising moderators of treatment outcome. Professionals involved in clinical practice agree relatively well on a variety of factors that influence outcome, including specific characteristics of child/adolescent cases (e.g., severity and duration of dysfunction, comorbidity, experience of distress) and their parents and families (e.g., parental involvement in treatment, parental cooperation, absence of parental dysfunction, stable home life; Kazdin et al., 1990). Several characteristics of the therapist (experience, training, relationship with the child) are also accorded a significant role in outcome and would be priorities for treatment research. In passing, it is worth noting that practitioners accord less significance to treatment technique as a determinant of outcome than to the variables listed previously. The integration of other variables than technique would seem to be a high priority for outcome research on several counts. Apart from the type of research, there may be value in incorporating characteristics of clinical treatment into research. The focus of research on relatively brief treatments, with nonreferred cases, in school settings, represents an important focus that need not be defended. At the same time, because so few studies focus on more extended treatments, in clinical settings, the generality of research findings are uncomfortably in question. Studies need not mimic clinical work to provide generalizable findings. Also, research by its very nature introduces conditions that simplify conditions of application to permit answers to critical questions. Merely because treatment research departs from clinical practice alone does not necessarily mean that the findings from research lack generality (Kazdin, 1978). However, it would be useful to learn, for example, whether the usual duration of treatment in clinical work exerts materially different impact from the highly abbreviated procedures that often characterize versions used in research, whether parent involvement critical in clinical work but less often examined in research, makes a difference, and so on. The differences between research and practice are not merely quantitative (e.g., more treatment, more severe cases) but qualitative as well (e.g., different treatments, types of cases). As the gap between research and practice increases on a given dimension, the degree of faith required to sustain clinical practice increases. The present review might well end with the suggestion that more research is needed. That would be an unfortunate communication of our purpose. Clearly, more research is needed. However, the focus and direction of research may warrant reconsideration. There remains a strong need to focus treatment to evaluate the oft-cited question and to place clinical work on firm empirical footing. The present study was designed to identify possible leads so that future research will retain closer ties to these goals.

References American Psychiatric Association. (1980). Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.

739

American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd ed., rev.). Washington, DC: Author. Barrett, C. L., Hampe, I. E., & Miller, L. C. (1978). Research on child psychotherapy. In S. L. Garfield & A. E. Bergin (Eds.), Handbook of psychotherapy and behavior change: An empirical analysis (2nd ed., pp. 411-435). New York: Wiley. Brown, J. (1987). A review of meta-analyses conducted on psychotherapy outcome research. Clinical Psychology Review, 7,1-23. Casey, R. J., & Berman, J. S. (1985). The outcome of psychotherapy with children. Psychological Bulletin, 98, 388-400. Cohen, J. (1988). Statistical power analyses for the behavioral sciences (2nd ed.). Hillsdale, NY: Erlbaum. Feldman, R. A., Caplinger, T. E., & Wodarski, J. S. (1983). The St. Louis conundrum: The effective treatment of antisocial youths. Englewood Cliffs, NJ: Prentice-Hall. Garfield, S. L. (1980). Psychotherapy: An eclectic approach. New York: Wiley. Garfield, S. L., & Bergin, A. E. (Eds.). (1986). Handbook of psychotherapy and behavior change (3rd ed.). New York: Wiley. Goldfried, M. R., Greenberg, L. S., & Marmar, C. (1990). Individual psychotherapy: Process and outcome. Annual Review of Psychology, 41, 659-688. Hedges, L. Y, & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Heinicke, C. M., & Strassmann, L. H. (1975). Toward more effective research on child psychotherapy. Journal of the American Academy of Child Psychiatry, 3, 561-588. Institute of Medicine. (1989). Research on children and adolescents with mental, behavioral, and developmental disorders. Washington, DC: National Academy Press. Kazdin, A. E. (1978). Evaluating the generality of findings in analogue therapy research. Journal of Consulting and Clinical Psychology, 46, 673-686. Kazdin, A. E. (1986). The evaluation of psychotherapy: Research designs and methodology. In S. L. Garfield & A. E. Bergin (Eds.), Handbook of'psychotherapy andbehaviorchange (3rd ed., pp. 23-68). New York: Wiley. Kazdin, A. E. (1988). Child psychotherapy: Developing and identifying effective treatments. Elmsford, NY: Pergamon Press. Kazdin, A. E. (1990). Psychotherapy for children and adolescents. Annual Review of Psychology, 41, 21-54. Kazdin, A. E., & Bass, D. (1989). Power to detect differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology. 57,138-147. Kazdin, A. E., Siegel, T. C, & Bass, D. (1990). Drawing upon clinical practice to inform research on child and adolescent psychotherapy: A survey of practitioners. Professional Psychology: Research and Practice, 21,189-198. Kiesler, D. J. (1971). Experimental designs in psychotherapy research. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change: An empirical analysis (pp. 36-74). New \brk: Wiley. Kolvin, I., Garside, R. F., Nicol, A. R., MacMillan, A., Wolstenholme, F., & Leitch, I. M. (1981). Help starts here: The maladjusted child in the ordinary school. London: Tavistock. Koocher, G. P., & Pedulla, B. M. (1977). Current practices in child psychotherapy. Professional Psychology. 8, 275-287. Kovacs, M., & Paulauskas, S. (1986). The traditional psychotherapies. In H. C. Quay & J. S. Werry (Eds.), Psychopathological disorders of childhood (3rd ed., pp. 496-522). New York: Wiley. Lambert, M. J., Shapiro, D. A., & Bergin, A. E. (1986). The effectiveness of psychotherapy. In S. L. Garfteld & A. E. Bergin (Eds.), Handbook of psychotherapy and behavior change (3rd ed., pp. 157-211). New York: Wiley.

This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

740

KAZD1N, BASS, AYERS, AND RODGERS

Paul, G. L. (1967). Outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109-118. Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646-656. Shaffer, D. (1984). Notes on psychotherapy research among children and adolescents. Journal of the American Academy of Child Psychiatry. 23,552-561. Silver, L. B., & Silver, B. J. (1983). Clinical practice of child psychiatry: A survey. Journal of the American Academy of Child Psychiatry, 22, 573-579. Smith, M. L., Glass, G. V, & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press. Tuma, J. M., & Pratt, J. M. (1982). Clinical child psychology practice and training: A survey. Journal of Clinical Child Psychology, 11, 2734.

Tuma, J. M., & Sobotka, K. R. (1983). Traditional therapies with children. In T. H. Ollendick & M. Hersen (Eds.), Handbook of child psychopathology (pp. 391-426). New York: Plenum Press. United States Congress, Office of Technology Assessment. (1986). Children's mental health: Problems and services—A background paper. Washington, DC: United States Government Printing Office. Waldron-Skinner, S. (1986). Dictionary of psychotherapy. London: Routledge & Regan Paul. Weisz, J. R., Weiss, B., Alicke, M. D, & Klotz, M. L. (1987). Effectiveness of psychotherapy with children and adolescents: Meta-analytic findings for clinicians. Journal ofConsulting andClinical Psychology, 55, 542-549. Received March 15,1990 Revision received May 21,1990 Accepted May 25,1990 •

Harvey Appointed Editor of Contemporary Psychology, 1992-1997 The Publications and Communications Board of the American Psychological Association announces the appointment of John H. Harvey, University of Iowa, as editor of Contemporary Psychology for a 6-year term beginning in 1992. Publishers should note that books should not be sent to Harvey. Publishers should continue to send two copies of books to be considered for review plus notices of publication to PsycINFO Services Department, APA Attn: Contemporary Psychology Processing 1400 North Uhle Street Arlington, Virginia 22201 Please note that all reviews are written by invitation.