This article was downloaded by: [University of Glasgow] On: 11 October 2014, At: 03:11 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study a
a
Mariola Moeyaert , Maaike Ugille , John M. Ferron b
, S. Natasha Beretvas & Wim Van den Noortgate
a
c
Katholieke Universiteit Leuven
b
University of South Florida
a
c
University of Texas Published online: 01 Oct 2013.
To cite this article: Mariola Moeyaert , Maaike Ugille , John M. Ferron , S. Natasha Beretvas & Wim Van den Noortgate (2013) The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study, Multivariate Behavioral Research, 48:5, 719-748, DOI: 10.1080/00273171.2013.816621 To link to this article: http://dx.doi.org/10.1080/00273171.2013.816621
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [University of Glasgow] at 03:11 11 October 2014
Multivariate Behavioral Research, 48:719–748, 2013 Copyright © Taylor & Francis Group, LLC ISSN: 0027-3171 print/1532-7906 online DOI: 10.1080/00273171.2013.816621
The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study Mariola Moeyaert and Maaike Ugille Katholieke Universiteit Leuven
John M. Ferron University of South Florida
S. Natasha Beretvas University of Texas
Wim Van den Noortgate Katholieke Universiteit Leuven
Previous research indicates that three-level modeling is a valid statistical method to make inferences from unstandardized data from a set of single-subject experimental studies, especially when a homogeneous set of at least 30 studies are included (Moeyaert, Ugille, Ferron, Beretvas, & Van den Noortgate, 2013a). When singlesubject data from multiple studies are combined, however, it often occurs that the dependent variable is measured on a different scale, requiring standardization of the data before combining them over studies. One approach is to divide the dependent variable by the residual standard deviation. In this study we use Monte Carlo methods to evaluate this approach. We examine how well the fixed effects (e.g., immediate treatment effect and treatment effect on the time trend) and the variance
Correspondence concerning this article should be addressed to Mariola Moeyaert, Faculty of Psychology and Educational Sciences, Katholieke Universiteit Leuven, Andreas Vesaliusstraat 2 – Box 3762, B-3000 Leuven, Belgium. E-mail: [email protected]
719
720
MOEYAERT ET AL.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
components (the between- and within-subject variance) are estimated under a number of realistic conditions. The three-level synthesis of standardized single-subject data is found appropriate for the estimation of the treatment effects, especially when many studies (30 or more) and many measurement occasions within subjects (20 or more) are included and when the studies are rather homogeneous (with small between-study variance). The estimates of the variance components are less accurate.
In a single-subject experimental design (SSED), the outcome variable of one subject is measured repeatedly within and across different conditions or phases (e.g., baseline phase or A-phase, treatment phase, or B-phase). Although the use of SSEDs has grown, systematic reviews and meta-analyses of treatment effects often include only studies using group-comparison studies to estimate changes between different conditions under investigation (Van den Noortgate & Onghena, 2008). The exclusion of SSEDs from these reviews is a matter of concern because information about the variation between subjects in the magnitude of treatment effects tends to be lost in group-comparison designs, which provide averages and effect sizes only for the entire group. A limitation of single-subject designs is that the corresponding results are subject specific and therefore not generalizable to other subjects. In order to address this problem, researchers can replicate single-subject experiments within studies (e.g., multiple-baseline designs). Among single-subject designs, these multiple-baseline designs are preferred (Shadish & Sullivan, 2011) because the staggering of the treatment across subjects makes it possible to disentangle real treatment effects from extraneous factors like maturation or history. As a result, these designs are increasingly popular, as shown in Figure 1.
FIGURE 1 Evolution of the number of citations in each year from 2000 to 2011 for the term “multiple-baseline” using the Social Science Citation Index (color figure available online).
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL SYNTHESIS
721
Another way to address the problem of generalizability is the replication of SSED across studies. Combining data of replicated SSEDs can, for instance, be accomplished by using a meta-analysis of effect sizes (Busk & Serlin, 1992; Maggin, Swaminathan, Rogers, O’Keeffe, Sugai, & Horner, 2011). A problem is that there is no consensus in the literature about the effect size metric to be used. A number of nonparametric effect size metrics have been proposed to analyze single-case designs (e.g., percentage of nonoverlapping data, percentage of all nonoverlapping data, or percentage exceeding the median). Although these nonparametric effect size measures for single-case research can be used without making distributional assumptions, they entail at least three weaknesses. First, such measures may be influenced by outliers in the baseline phase (Allison & Gorman, 1993; Salzberg, Strain, & Baer, 1987). A second drawback is their insensitivity to data trends and variability in the data (White, 1987; Wolery, Busick, Reichow, & Barton, 2010), and third, the sampling distributions of these metrics are unknown, which limits the validity of statistical tests such as moderator analyses that are often conducted in meta-analytic work (Beretvas & Chung, 2008). Other effect size measures are based on regression models (Beretvas & Chung, 2008; Van den Noortgate & Onghena, 2003). Treatment effects in single-case studies are further sometimes tested using nonparametric randomization tests. Randomization tests can also be used to test the existence of a treatment effect in a set of single-case studies (Edgington & Onghena, 2007). Up to now, the most common way to analyze single-case data is by using visual analyses (Hersen & Barlow, 1976; Kazdin, 1982; Kennedy, 2005; Kratochwill, 1978; Kratochwill & Levin, 1992; McReynolds & Kearns, 1983; Richards, Taylor, Ramasamy, & Richards, 1999; Tawney & Gast, 1984; White & Haring, 1980). Although visual analysis might do justice to the richness of single-case data, this method tends to result in too many Type I errors (Fisch, 2001; Normand & Bailey, 2006) and Type II errors (Jones, Weinrott, & Vaught, 1978; Ottenbacher, 1993). In this article we focus on a parametric statistical method to summarize single-case results over cases and over studies, namely, the three-level modeling of the raw data. Previous studies indicate that the three-level modeling is a valid statistical method to combine data (Ferron, Farmer, & Owens, 2010; Moeyaert et al., 2013a; Owens & Ferron, 2012; Shadish & Rindskopf, 2007; Van den Noortgate & Onghena, 2003, 2008). A complexity when single-subject data from multiple studies are combined is that the dependent variable often is not measured on a common scale, requiring a standardization of the data before combining them over studies. Given the importance of standardization, we discuss and evaluate in this study one specific standardization method used before combining the data by means of a threelevel model. In the following paragraphs, we first present the three-level model to aggregate single-subject data. Then, we describe the method to standardize
722
MOEYAERT ET AL.
single-subject data. Next, we present the setup and results of a Monte Carlo simulation study evaluating the analysis of the three-level modeling of standardized single-subject data.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL MODELING Hierarchical structures occur naturally: for instance, patients are clustered, nested, or grouped within clinics within health authorities; voters are nested within polling districts within constituencies; and citizens are grouped within cities within countries. Kreft and de Leeuw (1998) expressed this as follows: “Once you know that hierarchies exist you see them everywhere” (p. 1). Also data in social and behavioral sciences are usually characterized by a hierarchical structure and therefore require statistical analysis methods that account for this structure. Schooling systems present an obvious example of a hierarchical structure: students are grouped within classes that themselves are grouped within schools. We refer to a hierarchy as consisting of units grouped at different levels. In this example, students are the Level 1 units, classes the Level 2 units, and schools the Level 3 units in a three-level structure. A different example of hierarchically structured data occurs when the same case or subject is measured repeatedly within and across different conditions or phases (e.g., a baseline phase and a treatment phase), such as in SSEDs. If we have a set of studies in which one or a few subjects are investigated, we can see a three-level structure: measurement occasions at the first level are grouped within cases or subjects at the second level, which in turn are grouped in studies at the third level (see Figure 2). A three-level model can be used to analyze such a data structure. An advantage of the use of a three-level model is that it allows one to estimate withinsubject, between-subject, and between-study variance. Moreover, ignoring the study level would imply that we do not take into account that subjects from the same study are more alike than subjects from different studies. Van den Noortgate, Opdenakker, and Onghena (2005) showed that ignoring a top (or intermediate) level has significant effects on the results of a multilevel analysis using hierarchical linear models. At the first level of this three-level model, a regression equation describes the within-subject variability (Equation 1). Ytj k describes the score on the dependent variable on measurement occasion i .i D 1; 2; : : : ; I /, for subject j .j D 1; 2; : : : ; J / in study k .k D 1; 2; : : : ; K/ as a linear function of two predictors and their interaction, more specifically a time indicator .Tij k /, for instance the session number, and a dummy coded variable .Dij k / indicating whether the measurement occasion i from the j th subject in study k belongs to the baseline
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL SYNTHESIS
723
FIGURE 2 The three-level hierarchical structure for the synthesis of single-subject experimental data (color figure available online).
phase .Dij k D 0/ or the treatment phase .Dij k D 1/. Yij k D “0j k C “1j k Tij k C “2j k Dij k C “3j k Tij k Dij k C eij k with eij k N.0; ¢e2 /:
(1)
If the time indicator is coded such that it equals zero at the start of the treatment phase, “oj k indicates the expected baseline level at the start of the treatment phase (when Tij k D 0), “1j k is the linear time trend in the baseline scores, the coefficient “2j k is then the immediate effect of the treatment on the outcome, and “3j k refers to the effect of the intervention on the trend. The regression coefficients have indices j and k, meaning that they are subject and study specific. At the second level, the variation across subjects is modeled in the following four equations: 8 “0j k ˆ ˆ ˆ
Multivariate Behavioral Research Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hmbr20
The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study a
a
Mariola Moeyaert , Maaike Ugille , John M. Ferron b
, S. Natasha Beretvas & Wim Van den Noortgate
a
c
Katholieke Universiteit Leuven
b
University of South Florida
a
c
University of Texas Published online: 01 Oct 2013.
To cite this article: Mariola Moeyaert , Maaike Ugille , John M. Ferron , S. Natasha Beretvas & Wim Van den Noortgate (2013) The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study, Multivariate Behavioral Research, 48:5, 719-748, DOI: 10.1080/00273171.2013.816621 To link to this article: http://dx.doi.org/10.1080/00273171.2013.816621
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [University of Glasgow] at 03:11 11 October 2014
Multivariate Behavioral Research, 48:719–748, 2013 Copyright © Taylor & Francis Group, LLC ISSN: 0027-3171 print/1532-7906 online DOI: 10.1080/00273171.2013.816621
The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study Mariola Moeyaert and Maaike Ugille Katholieke Universiteit Leuven
John M. Ferron University of South Florida
S. Natasha Beretvas University of Texas
Wim Van den Noortgate Katholieke Universiteit Leuven
Previous research indicates that three-level modeling is a valid statistical method to make inferences from unstandardized data from a set of single-subject experimental studies, especially when a homogeneous set of at least 30 studies are included (Moeyaert, Ugille, Ferron, Beretvas, & Van den Noortgate, 2013a). When singlesubject data from multiple studies are combined, however, it often occurs that the dependent variable is measured on a different scale, requiring standardization of the data before combining them over studies. One approach is to divide the dependent variable by the residual standard deviation. In this study we use Monte Carlo methods to evaluate this approach. We examine how well the fixed effects (e.g., immediate treatment effect and treatment effect on the time trend) and the variance
Correspondence concerning this article should be addressed to Mariola Moeyaert, Faculty of Psychology and Educational Sciences, Katholieke Universiteit Leuven, Andreas Vesaliusstraat 2 – Box 3762, B-3000 Leuven, Belgium. E-mail: [email protected]
719
720
MOEYAERT ET AL.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
components (the between- and within-subject variance) are estimated under a number of realistic conditions. The three-level synthesis of standardized single-subject data is found appropriate for the estimation of the treatment effects, especially when many studies (30 or more) and many measurement occasions within subjects (20 or more) are included and when the studies are rather homogeneous (with small between-study variance). The estimates of the variance components are less accurate.
In a single-subject experimental design (SSED), the outcome variable of one subject is measured repeatedly within and across different conditions or phases (e.g., baseline phase or A-phase, treatment phase, or B-phase). Although the use of SSEDs has grown, systematic reviews and meta-analyses of treatment effects often include only studies using group-comparison studies to estimate changes between different conditions under investigation (Van den Noortgate & Onghena, 2008). The exclusion of SSEDs from these reviews is a matter of concern because information about the variation between subjects in the magnitude of treatment effects tends to be lost in group-comparison designs, which provide averages and effect sizes only for the entire group. A limitation of single-subject designs is that the corresponding results are subject specific and therefore not generalizable to other subjects. In order to address this problem, researchers can replicate single-subject experiments within studies (e.g., multiple-baseline designs). Among single-subject designs, these multiple-baseline designs are preferred (Shadish & Sullivan, 2011) because the staggering of the treatment across subjects makes it possible to disentangle real treatment effects from extraneous factors like maturation or history. As a result, these designs are increasingly popular, as shown in Figure 1.
FIGURE 1 Evolution of the number of citations in each year from 2000 to 2011 for the term “multiple-baseline” using the Social Science Citation Index (color figure available online).
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL SYNTHESIS
721
Another way to address the problem of generalizability is the replication of SSED across studies. Combining data of replicated SSEDs can, for instance, be accomplished by using a meta-analysis of effect sizes (Busk & Serlin, 1992; Maggin, Swaminathan, Rogers, O’Keeffe, Sugai, & Horner, 2011). A problem is that there is no consensus in the literature about the effect size metric to be used. A number of nonparametric effect size metrics have been proposed to analyze single-case designs (e.g., percentage of nonoverlapping data, percentage of all nonoverlapping data, or percentage exceeding the median). Although these nonparametric effect size measures for single-case research can be used without making distributional assumptions, they entail at least three weaknesses. First, such measures may be influenced by outliers in the baseline phase (Allison & Gorman, 1993; Salzberg, Strain, & Baer, 1987). A second drawback is their insensitivity to data trends and variability in the data (White, 1987; Wolery, Busick, Reichow, & Barton, 2010), and third, the sampling distributions of these metrics are unknown, which limits the validity of statistical tests such as moderator analyses that are often conducted in meta-analytic work (Beretvas & Chung, 2008). Other effect size measures are based on regression models (Beretvas & Chung, 2008; Van den Noortgate & Onghena, 2003). Treatment effects in single-case studies are further sometimes tested using nonparametric randomization tests. Randomization tests can also be used to test the existence of a treatment effect in a set of single-case studies (Edgington & Onghena, 2007). Up to now, the most common way to analyze single-case data is by using visual analyses (Hersen & Barlow, 1976; Kazdin, 1982; Kennedy, 2005; Kratochwill, 1978; Kratochwill & Levin, 1992; McReynolds & Kearns, 1983; Richards, Taylor, Ramasamy, & Richards, 1999; Tawney & Gast, 1984; White & Haring, 1980). Although visual analysis might do justice to the richness of single-case data, this method tends to result in too many Type I errors (Fisch, 2001; Normand & Bailey, 2006) and Type II errors (Jones, Weinrott, & Vaught, 1978; Ottenbacher, 1993). In this article we focus on a parametric statistical method to summarize single-case results over cases and over studies, namely, the three-level modeling of the raw data. Previous studies indicate that the three-level modeling is a valid statistical method to combine data (Ferron, Farmer, & Owens, 2010; Moeyaert et al., 2013a; Owens & Ferron, 2012; Shadish & Rindskopf, 2007; Van den Noortgate & Onghena, 2003, 2008). A complexity when single-subject data from multiple studies are combined is that the dependent variable often is not measured on a common scale, requiring a standardization of the data before combining them over studies. Given the importance of standardization, we discuss and evaluate in this study one specific standardization method used before combining the data by means of a threelevel model. In the following paragraphs, we first present the three-level model to aggregate single-subject data. Then, we describe the method to standardize
722
MOEYAERT ET AL.
single-subject data. Next, we present the setup and results of a Monte Carlo simulation study evaluating the analysis of the three-level modeling of standardized single-subject data.
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL MODELING Hierarchical structures occur naturally: for instance, patients are clustered, nested, or grouped within clinics within health authorities; voters are nested within polling districts within constituencies; and citizens are grouped within cities within countries. Kreft and de Leeuw (1998) expressed this as follows: “Once you know that hierarchies exist you see them everywhere” (p. 1). Also data in social and behavioral sciences are usually characterized by a hierarchical structure and therefore require statistical analysis methods that account for this structure. Schooling systems present an obvious example of a hierarchical structure: students are grouped within classes that themselves are grouped within schools. We refer to a hierarchy as consisting of units grouped at different levels. In this example, students are the Level 1 units, classes the Level 2 units, and schools the Level 3 units in a three-level structure. A different example of hierarchically structured data occurs when the same case or subject is measured repeatedly within and across different conditions or phases (e.g., a baseline phase and a treatment phase), such as in SSEDs. If we have a set of studies in which one or a few subjects are investigated, we can see a three-level structure: measurement occasions at the first level are grouped within cases or subjects at the second level, which in turn are grouped in studies at the third level (see Figure 2). A three-level model can be used to analyze such a data structure. An advantage of the use of a three-level model is that it allows one to estimate withinsubject, between-subject, and between-study variance. Moreover, ignoring the study level would imply that we do not take into account that subjects from the same study are more alike than subjects from different studies. Van den Noortgate, Opdenakker, and Onghena (2005) showed that ignoring a top (or intermediate) level has significant effects on the results of a multilevel analysis using hierarchical linear models. At the first level of this three-level model, a regression equation describes the within-subject variability (Equation 1). Ytj k describes the score on the dependent variable on measurement occasion i .i D 1; 2; : : : ; I /, for subject j .j D 1; 2; : : : ; J / in study k .k D 1; 2; : : : ; K/ as a linear function of two predictors and their interaction, more specifically a time indicator .Tij k /, for instance the session number, and a dummy coded variable .Dij k / indicating whether the measurement occasion i from the j th subject in study k belongs to the baseline
Downloaded by [University of Glasgow] at 03:11 11 October 2014
THREE-LEVEL SYNTHESIS
723
FIGURE 2 The three-level hierarchical structure for the synthesis of single-subject experimental data (color figure available online).
phase .Dij k D 0/ or the treatment phase .Dij k D 1/. Yij k D “0j k C “1j k Tij k C “2j k Dij k C “3j k Tij k Dij k C eij k with eij k N.0; ¢e2 /:
(1)
If the time indicator is coded such that it equals zero at the start of the treatment phase, “oj k indicates the expected baseline level at the start of the treatment phase (when Tij k D 0), “1j k is the linear time trend in the baseline scores, the coefficient “2j k is then the immediate effect of the treatment on the outcome, and “3j k refers to the effect of the intervention on the trend. The regression coefficients have indices j and k, meaning that they are subject and study specific. At the second level, the variation across subjects is modeled in the following four equations: 8 “0j k ˆ ˆ ˆ
