This article was downloaded by: [York University Libraries] On: 15 November 2014, At: 14:56 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
An Empirical Investigation of Partial Effect Sizes in MetaAnalysis of Correlational Data Ariel M. Aloe
a
a
University of Northern Iowa Published online: 20 Dec 2013.
To cite this article: Ariel M. Aloe (2014) An Empirical Investigation of Partial Effect Sizes in Meta-Analysis of Correlational Data, The Journal of General Psychology, 141:1, 47-64, DOI: 10.1080/00221309.2013.853021 To link to this article: http://dx.doi.org/10.1080/00221309.2013.853021
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [York University Libraries] at 14:56 15 November 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
The Journal of General Psychology, 2014, 141(1), 47–64 C 2014 Taylor & Francis Group, LLC Copyright
Downloaded by [York University Libraries] at 14:56 15 November 2014
An Empirical Investigation of Partial Effect Sizes in Meta-Analysis of Correlational Data ARIEL M. ALOE University of Northern Iowa
ABSTRACT. The partial correlation and the semi-partial correlation can be seen as measures of partial effect sizes for the correlational family. Thus, both indices have been used in the meta-analysis literature to represent the relationship between an outcome and a predictor of interest, controlling for the effect of other variables in the model. This article evaluates the accuracy of synthesizing these two indices under different situations. Both partial correlation and the semi-partial correlation appear to behave as expected with respect to bias and root mean squared error (RMSE). However, the partial correlation seems to outperform the semi-partial correlation regarding Type I error of the homogeneity test (Q statistic). Although further investigation is needed to fully understand the impact of meta-analyzing partial effect sizes, the current study demonstrates the accuracy of both indices. Keywords: meta-analysis, partial effect sizes, regression results
META-ANALYSIS IS “THE STATISTICAL ANALYSIS of a large collection of analysis results from individual studies for the purposes of integrating findings” (Glass, 1976, p. 3). In meta-analysis, study results are transformed to a common metric and represented as effect sizes for analysis. Effect sizes are scale-free indices that assess the magnitude and direction of the relationship between variables. Nearly all effect-size indices, used in meta-analysis, have variances that can be computed from sufficient statistics representing the uncertainty in the effect-size values. Thus, effect sizes are a crucial component of any meta-analysis. Partial effect sizes for the r-family (correlation coefficient) of effects have been formally discussed by Becker and Wu (2007) and Aloe and Becker (2012). However, none of these sources empirically investigate the accuracy of the estimations nor the behavior of the homogeneity test when meta-analyzing partial effect sizes. The two partial effect sizes considered in this article are the partial correlation and the semi-partial correlation. Along with Keef and Roberts (2004), Address correspondence to Ariel M. Aloe, University of Northern Iowa, 613 Schindler Education Center, Cedar Falls, IA 50614, USA; [email protected] (e-mail). 47
Downloaded by [York University Libraries] at 14:56 15 November 2014
48
The Journal of General Psychology
who discussed partial effects in the context of standardized mean differences, we also refer to zero-order and partial effects to distinguish between the correlation coefficient (zero-order effects) and partial and semi-partial correlations (partial effects). There is extensive literature discussing the meta-analysis of zero order correlations (e.g., Field, 2001; Hafdahl, 2009; Hittner & May, 2012; Hunter & Schmidt, 1990). Nevertheless, the literature discussing the synthesis of partial effect sizes for the r-family of effects is, at best, very limited. A possible explanation is that many researchers, when conducting meta-analysis, treat partial effect sizes as zero order correlations (e.g., Bowman, 2010; Lynn & McCall, 2000; Mausbach, Moore, Roesch, Cardenas, & Patterson, 2010). Another possibility is that metaanalysts assume the statistical properties of zero order effects, such as the bivariate correlation, will transfer directly to partial effect sizes. Almost two decades ago, researchers argued that results from regression models can be treated as effect size indices (e.g., Becker & Schram, 1994; Hedges, Laine, & Greenwald, 1994). In order to confidently synthesize partial effect sizes, a better understanding of the general statistical properties of the synthesis of such partial effects is needed. As of today, there is no article that examines the behavior of partial effect sizes for the r-family. Thus, in this article we seek to evaluate the accuracy of the synthesis of partial effect sizes under fixed and random effect models. In addition, we study the behavior of the homogeneity test when partial or semi-partial correlations are synthesized. The overall organization of this article is as follows. First, the distinction between bivariate and partial effect sizes is discussed. The next section describes the partial and semi- partial correlation with their respective variances. Then, a brief overview of meta-analytical models is presented. A simulation study is presented and the article finishes with a set of general conclusions and recommendations for the meta-analyst. Bivariate and Partial Effects Bivariate effect sizes represent the bivariate association between a predictor (or independent) variable, and an outcome (or dependent) variable (for example, the Pearson product- moment correlation coefficient, r). In contrast, partial effect sizes (i.e., regression coefficients, partial correlations, and semi-partial correlations) represent the relationship between a predictor variable and an outcome variable while controlling for the effects of one or more additional variables. When predictors are correlated, it can have a significant impact on both the magnitude and direction of partial effects. The presence of collinearity can inflate correlation estimates, and in some cases can even reverse the sign of the bivariate correlation and its counterpart regression coefficient, partial correlation, and semi-partial correlation (Hocking, 1983). Additionally, when the relationship between the focal predictor and the outcome variable increases as a function of
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
49
another predictor being added to the model, suppression may occur. The presence of suppression may increase or decrease the magnitude of partial effect sizes, and in some cases can also reverse the sign with respect the bivariate correlation. Therefore, bivariate effects and partial effects represent different parameters and thus should not be treated as the same, should not be expected to have exactly the same properties as zero order effects, nor should they be combined within the same data set with zero order effects. Lastly, common meta-analytical techniques (e.g., Cooper, Hedges, & Valentine, 2009) can be used to synthesize partial effect sizes. However, one must take into account that each partial effect size could potentially arise from a different regression model (i.e., different sets of variables may be controlled or partialed out in different primary studies). When this happens, each partial effect in a metaanalysis could be estimating a different population parameter. This suggests that in some circumstances the synthesis of partial correlations or semi-partial correlations may not be possible or wise. In other circumstances modeling partial correlations may be more informative than reporting an overall partial effect. Accordingly, a meta-analyst faces two possible situations. In the first case, all the studies included in a meta-analysis of partial effect sizes may control for the same variables. This is not very likely in practice. Alternatively, the studies included in a meta-analysis of partial effect sizes may control for different covariates, which is more likely to be the case in practice. This suggests that when analyzing a collection of partial-effect values, the reviewer needs to include predictor variables that reflect differences in model complexity and are likely to influence the sizes of the partial correlation. Notation and Model The population regression model with p predictors can be written as Yj = β 0 + β 1 Xj1 + β 2 Xj2 + · · · + β p Xjp + ej , where β 0 is the intercept, β p is the regression coefficient(slope) associated with the variable Xp , and ej is the error term that follows a normal distribution such that ej ∼ N(0, σ 2). For consistency, we will refer to the variable of interest, our focal predictor, as Xf . For the case with two predictors, the regression coefficient can be represented as a function of the bivariate correlations such as (ryf − ry2 rf2 )/ (1 − r2f2 ) where ryf , ry2 , rf2 are the bivariate correlations between the focal predictor and the outcome, between the other predictor in the model and the outcome, and between the two predictors, respectively. Next, we discuss the partial and semi-partial correlations. The Semi-Partial Correlation The semi-partial correlation (e.g., Nunnally, 1967, p. 155) or part correlation (e.g., McNemar, 1962, pp. 167–168) is the correlation of a variable, say X1 , with the outcome, Y, after the portion of Y that can be linearly predicted from
Downloaded by [York University Libraries] at 14:56 15 November 2014
50
The Journal of General Psychology
another variable, say X 2 , has been removed from Y. For instance, suppose we are studying how well teacher self-efficacy predicts teacher performance. We know that years of experience is an important factor determining both teacher selfefficacy and teacher performance. But we are only interested in the relationship between teacher performance and teacher self-efficacy uninfluenced by years of experience. Thus, we want to partial out years of experience from the correlation, while simultaneously not ruling out years of experience. Suppose we want years of experience partialed out of the teacher self-efficacy scores but not the teacher performance scores. Then years of experience can be partialed only from the teacher self-efficacy score by the semi-partial correlation. This definition can be clarified by its computational formula for a model with only two predictors. Let ry1 , ry2 , and r12 be the bivariate correlations between the outcome and covariate one, the outcome and covariate two, and between covariates one and two, respectively. Thus, the semi-partial correlation between the outcome and covariate one is ry1 − ry2 r12 rsp1 = 2 1 − r12
(1)
Given that typically the semi-partial correlation can be obtained from multiple regression results, Aloe and Becker (2012) proposed the use of the semi-partial correlation as a partial effect size in meta-analysis. Specifically, the semi-partial correlation can be computed as
rspf
tf 1 − RY2 =√ (n − p − 1)
(2)
where tf is the t test of the regression coefficient β f in the multiple regression model, is the total variance accounted for by the full model with p number of predictors, n − p − 1 are the degrees of freedom, and n is the number of cases in the analysis. The variance vi (rsp ) can be estimated as (Aloe & Becker, 2012) vi (rsp ) =
RY4 − 2RY2 + RY2 (f ) + 1 − RY4 (f ) n
(3)
where R2y(f) = R2y – r2sp . The Partial Correlation Now suppose that we are interested only in the relationship between teacher performance and teacher self-efficacy, uninfluenced by years of experience. This is to say, we want to partial out years of experience from the correlation. If the correlation between teacher self-efficacy and teacher performance was obtained
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
51
from a sample with all subjects having the same number of years of experience, we would have controlled for years of experience in our design. The partial correlation achieves this statistically (Kerlinger & Pedhazur, 1973, p. 84). The partial correlation is the correlation between teacher self-efficacy and teacher performance in a group in which years of experience is constant. If all the members of the group had the same number of years of experience, the correlation of teacher self-efficacy and teacher performance would be equivalent to the partial correlation. Introductory statistics books (Cohen & Cohen, 1975; Nunnally, 1967) define the partial correlation for the case with two predictors and one outcome as ry1 − ry2 r12 rp1 = 2 2 1 − ry2 1 − r12
(4)
The partial correlation can also be obtained for models with more than two predictors as tf rp1 = 2 tf + (n − p − 1)
(5)
where tf is the t test of the regression coefficient β f in the multiple regression model, n − p − 1 are the degrees of freedom, p is the number of predictors, and n is the number of cases in the analysis. The variance vi (rp ) can be estimated as (Anderson,1984) 2 1 − rp2 vi rp = n−p−1
(6)
Brief Overview of Meta-Analytical Models Let Ti . . . Tk be the estimates of partial effect sizes (i.e., partial or semipartial correlations) from k independent studies, each Ti is an estimate of θ i , the true partial effect size. Typically, ei is the estimate error (i.e., the amount that Ti deviates from θ i ), which is assumed to be normally distributed with a mean of zero and variance vi . The random-effects model acknowledges two sources of variation: within-study sampling error (vi ) and between-studies variability, denoted by τ 2. The random-effects model can be represented as Ti = μ + ui + ei ,
(7)
and the variance of Ti is equal to vi + τ 2. Thus, the differences between the estimated partial effect sizes and the overall mean, μ, includes a component of
Downloaded by [York University Libraries] at 14:56 15 November 2014
52
The Journal of General Psychology
sampling variation for each study (ei = Ti − θ i ) as well as a term for the differences between the population effect size (ui = θ i − μ). In the absence of betweenstudies heterogeneity, τ 2 = 0, the random-effects model reduces to the fixedeffectmodel, having one source of variation (within-study sampling error). For the reader interested in a more detailed discussion, see Borenstein and colleagues (2010) and Hedges and Vevea (1998). It is standard to test whether the effect-size estimates are homogeneous—i.e., if they are estimating a common population partial effect size, θ . Specifically, to test the hypothesis H 0 : θ i = θ , i,. .., k, we use (Hedges, 1983) QT =
k (Ti − T¯ )2 vi i=1
(8)
where T¯ is the inverse-variance weighted mean partial effect size and vi is the within-study sampling variance for the partial effect size from study i, which will differ according to the index use. If QT exceeds the critical value of the χ 2 distribution with k − 1 degrees of freedom, the null hypothesis is rejected (Hedges & Olkin, 1985). Thus, a statistically significant QT indicates that the partial effectsize estimates from our k studies do not share the same population effect size. In such cases, potentially explainable variation exists among the partial effect sizes. Simulation To study the empirical performance of partial correlations and semi-partial correlations as measures of partial effect sizes, a simulation study was performed. The simulation was programmed in R (R Core Team, 2012). The Cholesky decomposition was used to obtain the desired correlations among variables. For simulating each individual study’s rp and rsp values, multiple regression models with two continuous predictors and a continuous outcome were considered. The number of studies (k) per meta-analysis was 10, 20, 30, and 40. For each condition, 5,000 replications were performed. For each replication rp and rsp , as well as their variances, were estimated. To simulate data for each pseudo study, scores from a multivariate normal distribution were generated, with means μi = 0.0, and variances vi = 1. By manipulating the covariances σ ij , different levels of collinearity and relatedness of the predictors to the outcome were obtained. A random sample of size n, with an average of 200, was generated for each multivariate normal distribution. The correlations between the predictors and the outcome took on the values .2, .4, and .6, and the correlation between the predictors was manipulated through the correlation matrix such as ρ ij = −.2, .2, .4, and .8, depending on the condition. Specifically, data were generated so that for each set of k studies, an individual study’s partial effect size arose from the same values of correlations. First, a correlation matrix with three variables (i.e., three unique correlation
Aloe
53
coefficients) was created for each study with a population matrix as ⎞ 1 ρyf ρy2 ⎝ ρfy 1 ρf 2 ⎠ , ρ2y ρ2f 1
Downloaded by [York University Libraries] at 14:56 15 November 2014
⎛
Next, a linear model was created for each study as Yj = β 0 + β 1 Xj1 + β 2 Xj2 + ej , where β 1 = β f , which is the slope for the focal predictor. For each study, the t-test associated with the focal predictor was extracted and the partial effect size and its variance were estimated using the corresponding formulas. In addition, to generate a random-effects model, random variability was added into the correlation between the outcome and the focal predictor (i.e., ρ yf ). Five values of between-study variability were selected (τ 2 = .0025, .005, .01, .05, and .10). Results Table 1 reports bias for the weighted average partial and semi-partial correlations for different values of between-study variance (τ 2), different values of number of studies (k), and displays the true partial effect size values (ρ pf and ρ spf ). Results suggest that the weighted average partial correlation and semi-partial correlating are remarkably accurate for all the simulated conditions even for the smallest number of studies (i.e., k = 10). The bias of the weighted average partial effects appeared to be consistently larger for the largest value of population heterogeneity (τ 2 = .10) compared to the other conditions. In addition, for the majority of the τ 2 = .10 conditions, results suggest an underestimation of the true partial effects for both partial and semi-partial correlations. The results also suggest that even in the presence of cooperative suppression and collinearity, the accuracy of the results is remarkable. If we let ryf represent the correlation between our outcome (Y) and our focal predictor (Xf ), cooperative suppression (Cohen & Cohen, 1975, p. 90) (i.e., reciprocal suppression Conger, 1974) occurs when ρ f 2 < 0. For example, for τ 2 = .00 (fixed-effect model), k = 10, ρ yf = .4, ρ y2 = .2, and ρ f2 = −.2, the bias for the weighted average partial and semi-partial correlations are −.0002 and .0038, respectively. It also appears that when there is relatively large collinearity between the predictors (i.e., ρ f2 = .80) the point estimate has excellent accuracy. For example, the result under τ 2 = .00 and k = 10 for ρ yf = .6, ρ y2 = .4, and ρ f2 = .8 indicates that the bias is .0043 for the partial correlation and a bias of −0.0012 for the semi-partial correlation. Table 2 gives root mean square error (RMSE) for the weighted average partial and semi- partial correlations for different values of between-study variance (τ 2) and different values of number of studies (k). The results also suggest that the RMSE of the weighted average partial and semi-partial correlations consistently increases as values of population heterogeneity (τ 2) increase. However, the RMSE
.286
.044
.40 −.20
.40
.40
.163
.20
.449
.367
.20 −.20
.20
.80 −.200
.245
.20 −.20
ρ spf
.20
ρ f2
ρ y2
ρ yf
k
.250 10 20 30 40 .167 10 20 30 40 .312 10 20 30 40 .048 10 20 30 40 −.218 10 20 30 40 .458 10 20 30 40 .375 10 20 30 40
ρ pf
.0001 .0005 .0005 .0006 .0003 .0003 .0001 .0000 −.0008 −.0006 −.0007 −.0003 −.0004 −.0006 .0002 .0001 .0004 .0003 .0004 .0004 −.0002 .0008 .0006 .0007 −.0001 .0011 .0009 .0007
r¯ spf .0027 .0034 .0035 .0036 .0021 .0023 .0022 .0022 .0031 .0040 .0040 .0045 .0002 .0001 .0011 .0009 .0026 .0031 .0031 .0032 .0038 .0053 .0053 .0055 .0034 .0052 .0050 .0049
r¯ pf
τ 2 = .00
.0012 .0008 .0007 .0010 .0007 .0011 .0004 .0004 −.0003 .0001 .0005 .0004 .0003 .0004 .0002 .0000 −.0010 −.0003 .0000 −.0006 .0016 .0024 .0022 .0017 .0023 .0014 .0017 .0012
r¯ spf .0040 .0038 .0038 .0040 .0027 .0032 .0026 .0025 .0037 .0044 .0048 .0048 .0010 .0012 .0010 .0007 .0039 .0033 .0030 .0037 .0056 .0065 .0064 .0059 .0061 .0053 .0056 .0052
r¯ pf
τ 2 = .0025
TABLE 1. Bias for Partial and Semi-Partial Correlations
.0010 .0007 .0010 .0008 .0005 .0002 .0007 .0004 .0002 .0002 −.0002 .0003 .0003 −.0002 .0002 .0002 −.0002 −.0003 −.0003 −.0005 .0013 .0014 .0019 .0016 .0010 .0009 .0012 .0013
r¯ spf .0037 .0037 .0041 .0039 .0024 .0023 .0028 .0026 .0043 .0047 .0045 .0051 .0010 .0006 .0010 .0010 .0032 .0035 .0036 .0037 .0053 .0058 .0063 .0060 .0047 .0049 .0053 .0055
r¯ pf
τ 2 = .005 r¯ pf
r¯ spf
r¯ pf
τ 2 = .05 r¯ spf
r¯ pf
(Continued on next page)
.0006 .0020 .0028 .0012 .0015 −.0008 .0014 .0002 −.0024 .0011 −.0010 −.0003 .0025 −.0009 .0008 .0009 −.0163 −.0183 −.0180 −.0186 −.0032 −.0023 −.0030 −.0031 −.0002 −.0015 −.0008 .0000
τ 2 = .10
.0008 .0035 .0004 .0025 −.0009 .0007 .0036 .0015 .0038 .0004 .0005 .0034 .0011 .0034 .0012 .0006 .0035 .0016 .0038 −.0004 .0016 .0035 .0015 .0030 .0004 .0007 .0028 −.0013 .0003 −.0020 .0009 .0029 .0000 .0016 .0002 .0003 .0024 .0009 .0026 −.0009 .0001 .0039 .0020 .0051 −.0038 .0012 .0054 −.0004 .0025 −.0007 .0000 .0040 .0009 .0040 −.0026 .0005 .0046 .0008 .0039 −.0019 −.0016 −.0011 −.0011 −.0006 .0018 .0008 .0016 −.0001 .0004 −.0012 −.0002 .0005 −.0001 .0005 .0003 −.0001 .0006 .0001 .0006 .0003 −.0013 .0039 .0030 −.0021 .0142 .0000 .0027 .0019 −.0008 .0162 −.0005 .0033 .0025 −.0015 .0158 .0001 .0026 .0033 −.0024 .0163 .0021 .0058 .0012 .0040 −.0050 .0020 .0060 .0025 .0054 −.0042 .0017 .0056 .0006 .0035 −.0049 .0017 .0057 .0020 .0049 −.0049 .0020 .0055 .0007 .0034 −.0023 .0013 .0050 .0015 .0043 −.0034 .0013 .0050 .0014 .0041 −.0028 .0012 .0050 .0009 .0037 −.0020
r¯ spf
τ 2 = .01
Downloaded by [York University Libraries] at 14:56 15 November 2014
54 The Journal of General Psychology
.60
ρ yf
.40
.694
.480
.467
.40
.80
.572
.20
−.20
.653
.133
.80
−.20
.262
.40
.20
.490
−.20
.40
ρ spf
ρ f2
ρ y2
.509
.524
.757
.583
.667
.145
.286
.535
ρ pf
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
k
−.0012 −.0012 −.0008 −.0008 −.0005 −.0005 −.0004 −.0005 −.0001 −.0002 −.0002 .0000 −.0002 .0005 .0005 .0005 .0003 .0008 .0005 .0008 −.0010 −.0011 −.0012 −.0009 −.0011 −.0011 −.0008 −.0009 −.0012 −.0012 −.0010 −.0008
r¯ spf .0045 .0052 .0057 .0059 .0033 .0037 .0040 .0040 .0020 .0020 .0022 .0025 .0042 .0053 .0056 .0057 .0047 .0056 .0056 .0060 .0040 .0045 .0046 .0050 .0044 .0052 .0057 .0058 .0043 .0049 .0054 .0058
r¯ pf
τ 2 = .00
.0010 .0008 .0008 .0005 −.0003 .0003 .0004 .0005 .0002 .0004 .0010 .0002 .0033 .0032 .0033 .0035 .0018 .0019 .0028 .0023 .0017 .0020 .0014 .0019 .0009 .0008 .0007 .0009 .0008 .0019 .0012 .0008
r¯ spf .0062 .0062 .0063 .0060 .0034 .0043 .0045 .0046 .0022 .0025 .0031 .0022 .0064 .0062 .0063 .0066 .0055 .0057 .0066 .0061 .0051 .0052 .0047 .0053 .0060 .0060 .0061 .0064 .0055 .0068 .0061 .0056
r¯ pf
τ 2 = .0025
.0003 .0008 .0001 .0002 .0003 .0001 .0000 −.0003 −.0004 .0005 −.0001 −.0003 .0019 .0028 .0027 .0029 .0014 .0024 .0024 .0028 .0013 .0014 .0015 .0016 .0003 .0004 .0003 .0006 .0014 .0008 .0010 .0009
r¯ spf .0057 .0067 .0060 .0061 .0043 .0043 .0043 .0040 .0015 .0027 .0021 .0019 .0053 .0061 .0060 .0062 .0053 .0065 .0064 .0069 .0050 .0052 .0052 .0055 .0057 .0062 .0061 .0065 .0064 .0061 .0063 .0062
r¯ pf
τ 2 = .005
TABLE 1. Bias for Partial and Semi-Partial Correlations (Continued)
.0012 .0014 .0013 .0011 .0008 −.0001 .0006 .0005 .0020 .0002 .0008 .0002 .0034 .0029 .0034 .0032 .0022 .0018 .0023 .0028 .0019 .0018 .0016 .0017 .0009 .0009 .0009 .0015 .0001 .0013 .0007 .0009
r¯ spf .0059 .0064 .0063 .0062 .0044 .0037 .0044 .0043 .0040 .0020 .0028 .0021 .0060 .0056 .0061 .0059 .0057 .0053 .0058 .0063 .0046 .0045 .0044 .0045 .0057 .0057 .0059 .0065 .0042 .0055 .0049 .0051
r¯ pf
τ 2 = .01
−.0025 −.0021 −.0011 −.0007 .0001 −.0008 .0005 .0004 −.0053 −.0019 −.0010 −.0017 −.0064 −.0040 −.0049 −.0040 .0005 .0000 −.0009 −.0010 −.0161 −.0152 −.0160 −.0150 −.0025 −.0023 −.0020 −.0028 −.0203 −.0190 −.0210 −.0221
r¯ spf .0003 .0009 .0020 .0025 .0027 .0018 .0032 .0032 −.0045 −.0008 .0001 −.0006 −.0049 −.0024 −.0033 −.0023 .0029 .0025 .0017 .0015 −.0178 −.0166 −.0174 −.0164 .0005 .0008 .0013 .0004 −.0179 −.0165 −.0187 −.0199
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
−.0101 −.0115 −.0112 −.0104 −.0024 −.0035 −.0034 −.0023 −.0148 −.0119 −.0117 −.0123 −.0247 −.0226 −.0221 −.0227 −.0117 −.0131 −.0129 −.0138 −.0404 −.0401 −.0404 −.0400 −.0139 −.0167 −.0172 −.0155 −.0612 −.0586 −.0605 −.0608
r¯ spf −.0098 −.0115 −.0110 −.0102 −.0007 −.0018 −.0017 −.0006 −.0141 −.0109 −.0106 −.0113 −.0242 −.0222 −.0216 −.0222 −.0099 −.0111 −.0111 −.0119 −.0472 −.0467 −.0470 −.0466 −.0124 −.0154 −.0160 −.0141 −.0603 −.0577 −.0598 −.0601
r¯ pf
τ 2 = .10
Aloe 55
.40
.20
.20
.20
.40
ρ y2
ρ yf
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
−.20
.20
−.20
.80
.40
−.20
.20
k
ρ f2
.0218 .0151 .0121 .0107 .0225 .0156 .0125 .0109 .0198 .0140 .0113 .0098 .0216 .0149 .0121 .0106 .0208 .0144 .0117 .0099 .0184 .0128 .0104 .0090 .0203 .0141 .0113 .0099
r¯ spf
.0225 .0158 .0129 .0116 .0232 .0163 .0131 .0114 .0216 .0155 .0128 .0115 .0238 .0165 .0134 .0118 .0229 .0160 .0131 .0113 .0190 .0139 .0118 .0106 .0210 .0152 .0125 .0112
r¯ pf
τ 2 = .00
.0331 .0227 .0181 .0160 .0329 .0225 .0187 .0161 .0309 .0213 .0176 .0149 .0340 .0233 .0188 .0163 .0445 .0303 .0254 .0214 .0303 .0206 .0172 .0145 .0315 .0215 .0178 .0150
r¯ spf .0342 .0237 .0190 .0169 .0340 .0234 .0195 .0168 .0340 .0237 .0198 .0170 .0376 .0258 .0209 .0180 .0490 .0335 .0281 .0239 .0313 .0218 .0185 .0158 .0326 .0225 .0189 .0162
r¯ pf
τ 2 = .0025
TABLE 2. RMSE for Partial and Semi-Partial Correlations
.0278 .0190 .0156 .0133 .0283 .0193 .0160 .0136 .0264 .0180 .0144 .0127 .0278 .0196 .0158 .0135 .0349 .0242 .0196 .0164 .0253 .0174 .0143 .0121 .0264 .0182 .0148 .0125
r¯ spf .0288 .0199 .0166 .0142 .0292 .0200 .0167 .0143 .0291 .0202 .0164 .0148 .0308 .0217 .0175 .0150 .0385 .0269 .0218 .0184 .0262 .0185 .0157 .0135 .0273 .0192 .0160 .0138
r¯ pf
τ 2 = .005
.0404 .0281 .0230 .0196 .0409 .0280 .0223 .0196 .0391 .0266 .0220 .0190 .0421 .0287 .0238 .0205 .0594 .0410 .0329 .0284 .0392 .0270 .0216 .0187 .0396 .0277 .0217 .0193
r¯ spf .0417 .0291 .0239 .0205 .0423 .0290 .0231 .0203 .0429 .0297 .0244 .0213 .0465 .0318 .0264 .0227 .0654 .0452 .0364 .0314 .0403 .0281 .0227 .0198 .0408 .0287 .0227 .0203
r¯ pf
τ 2 = .01
.0788 .0545 .0433 .0381 .0796 .0561 .0439 .0383 .0788 .0540 .0445 .0373 .0835 .0593 .0467 .0400 .1237 .0846 .0691 .0595 .0783 .0539 .0422 .0377 .0791 .0546 .0432 .0379
r¯ spf .1077 .0743 .0608 .0522 .1091 .0759 .0614 .0524 .1071 .0736 .0589 .0519 .1150 .0797 .0650 .0564 .1646 .1116 .0917 .0785 .1059 .0728 .0589 .0498 .1084 .0734 .0599 .0514
r¯ spf .1104 .0762 .0623 .0536 .1120 .0779 .0630 .0538 .1170 .0805 .0643 .0567 .1263 .0876 .0714 .0620 .1799 .1220 .1003 .0859 .1078 .0742 .0599 .0507 .1109 .0750 .0613 .0525
r¯ pf
τ 2 = .10
(Continued on next page)
.0809 .0561 .0446 .0393 .0819 .0577 .0452 .0395 .0865 .0593 .0489 .0410 .0921 .0653 .0514 .0441 .1355 .0927 .0756 .0651 .0798 .0552 .0431 .0387 .0810 .0560 .0443 .0389
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
56 The Journal of General Psychology
.60
ρ yf
.40
.20
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
−.20
.40
.80
.40
−.20
.20
−.20
.80
.40
k
ρ f2
ρ y2
.0170 .0118 .0093 .0082 .0206 .0138 .0111 .0097 .0214 .0148 .0119 .0102 .0135 .0094 .0076 .0065 .0158 .0109 .0087 .0078 .0123 .0086 .0069 .0059 .0171 .0119 .0096 .0084 .0173 .0119 .0096 .0081
r¯ spf
.0177 .0130 .0110 .0102 .0225 .0155 .0127 .0113 .0236 .0164 .0133 .0115 .0138 .0106 .0093 .0085 .0164 .0122 .0104 .0098 .0108 .0082 .0073 .0069 .0179 .0132 .0112 .0103 .0183 .0131 .0111 .0102
r¯ pf
τ 2 = .00
.0297 .0202 .0164 .0140 .0324 .0223 .0183 .0157 .0448 .0308 .0250 .0216 .0284 .0193 .0159 .0140 .0290 .0198 .0162 .0142 .0271 .0190 .0152 .0131 .0313 .0218 .0172 .0147 .0427 .0296 .0237 .0204
r¯ spf .0322 .0223 .0186 .0160 .0357 .0249 .0206 .0179 .0495 .0341 .0277 .0240 .0288 .0199 .0167 .0150 .0296 .0206 .0174 .0154 .0281 .0200 .0162 .0142 .0341 .0241 .0194 .0169 .0465 .0326 .0263 .0228
r¯ pf
τ 2 = .0025
.0240 .0164 .0134 .0114 .0275 .0185 .0154 .0129 .0349 .0240 .0196 .0169 .0220 .0152 .0126 .0111 .0235 .0161 .0134 .0118 .0214 .0146 .0119 .0102 .0251 .0174 .0142 .0119 .0322 .0224 .0181 .0159
r¯ spf .0260 .0184 .0153 .0135 .0304 .0207 .0173 .0147 .0386 .0267 .0218 .0188 .0224 .0160 .0136 .0123 .0241 .0172 .0147 .0134 .0217 .0154 .0127 .0114 .0271 .0194 .0162 .0141 .0351 .0247 .0204 .0182
r¯ pf
τ 2 = .005
TABLE 2. RMSE for Partial and Semi-Partial Correlations (Continued)
.0380 .0262 .0214 .0184 .0419 .0288 .0228 .0202 .0587 .0404 .0338 .0292 .0369 .0253 .0210 .0181 .0380 .0259 .0209 .0186 .0361 .0248 .0201 .0173 .0399 .0277 .0223 .0195 .0579 .0394 .0325 .0279
r¯ spf .0412 .0288 .0238 .0206 .0462 .0318 .0254 .0226 .0649 .0446 .0374 .0323 .0372 .0258 .0217 .0188 .0387 .0266 .0217 .0196 .0380 .0261 .0214 .0187 .0434 .0303 .0247 .0219 .0631 .0431 .0357 .0306
r¯ pf
τ 2 = .01
.0766 .0522 .0424 .0362 .0832 .0580 .0462 .0398 .1256 .0849 .0683 .0584 .0731 .0505 .0417 .0349 .0761 .0514 .0420 .0358 .0693 .0492 .0414 .0357 .0804 .0542 .0447 .0390 .1162 .0796 .0661 .0589
r¯ spf .0830 .0565 .0459 .0393 .0914 .0637 .0508 .0438 .1380 .0932 .0750 .0642 .0737 .0508 .0419 .0352 .0772 .0523 .0427 .0363 .0736 .0525 .0441 .0382 .0875 .0589 .0486 .0424 .1270 .0865 .0714 .0633
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
.1018 .0713 .0577 .0496 .1141 .0801 .0631 .0536 .1588 .1118 .0911 .0791 .0983 .0699 .0586 .0509 .1007 .0704 .0572 .0492 .0961 .0734 .0634 .0581 .1061 .0767 .0618 .0536 .1585 .1144 .1014 .0928
r¯ spf .1101 .0770 .0622 .0534 .1251 .0877 .0690 .0587 .1744 .1225 .0997 .0866 .0993 .0706 .0590 .0512 .1022 .0713 .0578 .0496 .1037 .0803 .0701 .0646 .1155 .0832 .0667 .0577 .1724 .1229 .1080 .0980
r¯ pf
τ 2 = .10
Aloe 57
58
The Journal of General Psychology
TABLE 3. Type I Error for Partial and Semi-Partial Correlations
Downloaded by [York University Libraries] at 14:56 15 November 2014
k = 10
k = 20
k = 30
k = 40
ρ yf
ρ y2
ρ f2
r¯ spf
r¯ pf
r¯ spf
r¯ pf
r¯ spf
r¯ pf
r¯ spf
r¯ pf
.20
.20
−.20 .20 −.20 .40 .80 −.20 .20 −.20 .40 .80 −.20 .20 −.20 .40 .80
.054 .058 .090 .131 .102 .022 .034 .038 .090 .118 .002 .006 .006 .038 .041
.062 .058 .058 .066 .059 .061 .060 .055 .059 .064 .057 .053 .058 .056 .057
.057 .072 .115 .172 .138 .016 .034 .035 .120 .162 .000 .003 .003 .038 .038
.070 .070 .067 .066 .069 .061 .068 .066 .069 .069 .053 .060 .059 .065 .060
.054 .075 .139 .212 .166 .012 .021 .031 .138 .179 .000 .001 .001 .032 .037
.067 .072 .080 .076 .071 .067 .061 .067 .069 .066 .062 .066 .059 .063 .059
.054 .074 .156 .250 .189 .010 .022 .024 .152 .227 .000 .001 .000 .035 .038
.070 .072 .087 .076 .077 .068 .070 .064 .065 .065 .068 .066 .055 .075 .061
.40 .40
.20 .40
.60
.20 .40
values clearly decrease as k increases, which is not surprising because as k increases, we have more data with which to make inf ferences about the model and its quality. Again, the results suggests that these indices behave as expected even in presence of collinearity. For example, the result for τ 2 = .00, k = 10, ρ yf = .6, ρ y2 = .4, and ρ f2 = .8 indicates that the RMSEs are .0173 and .0183 for rspf and rpf , respectively. These values are comparable to other conditions holding τ 2 and k constant. However, the results suggest that RMSE is consistently larger for conditions with a larger amount of collinearity than conditions in which ρ f2 is less than .80. Table 3 displays empirical Type I error rates of the Q test for both partial effect sizes. The empirical rejection reported in Table 3 is an indication of how well the Q test detects heterogeneity. When τ 2 = .00, each independent rpf and rspf were generated from the same patterns of correlations with identical numbers of predictors. Given that 5,000 replications were performed, the 95% confidence interval for the proportions of Q values greater than their corresponding1 critical value was expected to be .05 ± 1.96(.05(1 − .05)/ 5000)1/2 = .05 ± .006 = [.044, .056]. However, for the semi-partial correlation, Type I error rates were significantly higher or significantly lower than their nominal value for all conditions considered. On the other hand, for the partial correlation the results suggest that the rejection rate is close to nominal for the majority of the conditions, but somewhat higher
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
59
than the nominal value. This aligns with the findings of Alexander, Scozzaro, and Borodkin(1989) who reported higher than nominal Type I error rates for test of homogeneity of bivariate correlations. In addition, studying the behavior of the Q test for bivariate correlations, Viechtbauer (2007) also noticed that when k increases, the probability of rejecting the null hypothesis increases as well. This finding is also true in the current study of the partial correlation as a measure of partial effect size. Currently, there are no other simulation studies reporting on Type I error rates for the semi-partial correlation or partial correlation. However, higher than nominal Type I error rates for test of homogeneity of bivariate correlations were previously reported (Alexander et al., 1989; S´anchez-Meca & Mar´ın-Mart´ınez, 1997; Viechtbauer, 2007). Yet, the lower than nominal Type I error rates are unexpected, suggesting that the results of the Q test, when synthesizing semi-partial correlations may not be that informative. The empirical rejection rates are reported in Figure 1 for the partial correlation and Figure 2 for the semi-partial correlation. Both represent the rejection rate as a function of τ 2 and k for different combinations of correlation values. Our findings suggest that increasing τ 2 and/or k results in a higher probability of detecting heterogeneity. Similar results also were reported in studies that use the bivariate correlationas an effect size measure (e.g., Viechtbauer, 2007). The results also indicate that in presence of collinearity (see Figure 1 and 2: m, n, and o) the null appears to be rejected much sooner even for small k values (i.e., k = 10) than for conditions with smaller ρ f2 values. Finally, the true magnitude of the partial correlation and semi-partial correlation appear to have an effect on the rejection rate. For instance, comparing in Figure 1 condition (f), ρ pf = .757, and condition (l), ρ pf = .524, the probability of detecting heterogeneity appears higher for all k values. Last, attention should be given to the true value of the partial effect sizes (column 4 and 5 in Table 1). It is well known that the true magnitude of the partial correlation and semi-partial correlation are affected by the values of the correlation matrix and the number of predictors in the model. However, under the simplest condition the bias and RMSE of the estimates is quite remarkable, making the partial correlation and semi-partial correlation promising partial effect sizes. Discussion and Practical Recommendations We have discussed the partial correlation and semi-partial correlation as a measure of partial effect size for the r-family. Overall, the results of demonstrate that the weighted average partial correlation and semi-partial correlation are accurate for all the simulated conditions, even for the smallest number of studies and collinearity between the predictors. In addition, results suggest that the rejection rate of the homogeneity test is close to, but somewhat higher than, the nominal for the majority of the conditions for the partial correlation. However, the use of the homogeneity test for the semi-partial correlation is questionable.
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
0
●
●
●
●
●
●
(a)
(d)
●
(g)
●
(j)
●
●
.0025 .005
●
.01
(m)
●
.05
●
ρY1 = .2,ρY2 = .4,ρ12 = .8
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
●
(b)
●
(e)
●
●
(h)
●
(k)
●
●
.01
(n)
●
.05
●
Amount of Heterogeneity
.0025 .005
●
ρY1 = .4,ρY2 = .4,ρ12 = .8
●
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .4,ρY2 = .2,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
(c)
●
●
●
(f)
●
●
(i)
●
●
●
(l)
●
●
.0025 .005
●
.01
(o)
●
.05
●
ρY1 = .6,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .6,ρY2 = .4,ρ12 = .4
●
●
ρY1 = .6,ρY2 = .2,ρ12 = −.2
●
ρY1 = .6,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .2,ρ12 = .2
Downloaded by [York University Libraries] at 14:56 15 November 2014
FIGURE 1. Rejection rate of the Q test for partial correlations.
RR
.10
●
●
●
●
●
k ●
40
30
20
10
60 The Journal of General Psychology
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
0
●
●
●
●
●
●
(a)
(d)
●
(g)
●
●
(j)
●
.0025 .005
●
●
.01
(m)
●
.05
●
ρY1 = .2,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .2,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
●
(b)
●
(e)
●
(h)
●
●
(k)
●
.01
(n)
●
.05
●
Amount of Heterogeneity
.0025 .005
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .4,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .4,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .2,ρ12 = .2
FIGURE 2. Rejection rate of the Q test for semi-partial correlations.
RR
.10
●
●
●
●
●
0
●
●
●
●
●
(c)
●
●
(f)
●
●
(i)
●
●
(l)
●
●
.0025 .005
●
.01
(o)
●
.05
●
ρY1 = .6,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .6,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .6,ρY2 = .2,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .2,ρ12 = .2
Downloaded by [York University Libraries] at 14:56 15 November 2014
.10
●
●
●
●
●
k ●
40
30
20
10
Aloe 61
Downloaded by [York University Libraries] at 14:56 15 November 2014
62
The Journal of General Psychology
The overall results suggest that the meta-analysis of the partial and semipartial correlations appear to have desired properties regarding bias and accuracy. However, there are several limitations in the synthesis of partial effect size indices that meta-analysts should take into account. First, one must follow the inclusion criteria of meta-analysis. The studies included and the treatment of the data need to align with the inclusion criteria. If the inclusion criteria are set to synthesize bivariate associations, no measure of partial association should be included (i.e., semi-partial correlation or partial correlation). Thus, if the meta-analyst is planning to allow for measures of partial association to be included in their syntheses, this should be clearly stated in their inclusion criteria section and formulas used should be provided. It is noteworthy that having different sets of predictors leads to different interpretations of the partial and semi-partial correlation for each model. Thus, when interpreting overall results the meta-analyst should attempt to interpret the results while taking into account what has been partialed out. This can be extremely difficult since, in the social sciences, sometimes we find models that have more than 20 predictors (e.g., Chaney, 1995). A possible alternative is to take into account the larger constructs that the variables represent, not the specific predictors. For instance, in a meta-analysis of student academic achievement and teacher expectations, studies may control for different sets of variables, but all of the studies to some degree might control for students, teacher, and/or school characteristics. The decision about which partial effect size to use should be made by taking into account the research question, the interpretation that the meta-analyst wishes to achieve, and what information is reported in the primary studies. For instance, if a primary study failed to report the total variance explained (R2y ), the semipartial correlation could not be computed for this study. Lastly, partial effects can be larger or smaller than their corresponding bivariate correlations. Thus, it is recommended that meta-analysts do not treat partial effect sizes as bivariate effects. It is also advisable that meta-analyst do not combine, in the same data set, bivariate correlations with partial effects. AUTHOR NOTE Ariel M. Aloe is an assistant professor in the Department of Educational Psychology and Foundations, University of Northern Iowa. He specializes in research synthesis, meta-analysis, quantitative methods, and teacher quality. ACKNOWLEDGMENTS The author is grateful to Betsy Becker and Christopher Thompson for their feedback in an earlier version of this article.
Aloe
63
FUNDING This research has been partially supported by a grant from the National Science Foundation (NSF 1252263). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Downloaded by [York University Libraries] at 14:56 15 November 2014
REFERENCES Alexander, R. A., Scozzaro, M. J., & Borodkin, L. J. (1989). Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis. Psychological Bulletin, 106(2), 329–331. Aloe, A. M., & Becker, B. J. (2012). An effect size for regression predictors in metaanalysis. Journal of Educational and Behavioral Statistics, 37(2), 278–297. Anderson, T. W. (1984). An introduction to multivariate statistical analysis (2nd ed.). New York: Wiley. Becker, B. J., & Schram, C. M. (1994). Examining explanatory models through research synthesis. In The handbook of research synthesis (pp. 357–381). New York: Russell Sage Foundation. Becker, B. J., & Wu, M.-J. (2007). The synthesis of regression slopes in meta-analysis. Statistical Science, 22(3), 414–429. doi:10.1214/07-STS243 Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. Bowman, N. A. (2010). College diversity experiences and cognitive development: A metaanalysis. Review of Educational Research, 80(1), 4–33. Chaney, B. (1995). Student outcomes and the professional preparation of eight-grade teachers in science and mathematics: NSF/NELS. Rockville, MD: Westat. Cohen, J., & Cohen, P. (1975). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum. Conger, A. J. (1974). A revised definition for suppressor variables: A guide to their identification and interpretation. Educational & Psychological Measurement, 34(1), 35–46. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation. Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6(2), 161–180. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Research, 5(10), 3–8. Hafdahl, A. R. (2009). Improved fisher z estimators for univariate random-effects metaanalysis of correlations. British Journal of Mathematical and Statistical Psychology, 62(2), 233–261. Hedges, L. V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93(2), 388–395. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504. Hedges, L. V., Laine, R. D., & Greenwald, R. (1994). An exchange: part I: Does money matter? A meta-analysis of studies of the effects of differential school inputs on student outcomes. Educational Researcher, 23(3), 5–14.
Downloaded by [York University Libraries] at 14:56 15 November 2014
64
The Journal of General Psychology
Hittner, J. B., & May, K. (2012). How accurate is the pearson r-from-z approximation: A Monte Carlo stimulation study. Journal of General Psychology, 139(2), 68–77. Hocking, R. R. (1983). Developments in linear regression methodology: 1959–1982. Technometrics, 25(3), 219–245. Retrieved from http://www.jstor.org/stable/1268603 Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage Publications, Inc. Keef, S. P., & Roberts, L. A. (2004). The meta-analysis of partial effect sizes. British Journal of Mathematical & Statistical Psychology, 57(1), 97–129. Kerlinger, F., & Pedhazur, E. (1973). Multiple regression in behavioral research. New York, NY: Holt, Rinehart, and Winston. Lynn, M., & McCall, M. (2000). Gratitude and gratuity: A meta-analysis of research on the service-tipping relationship. Journal of Socio-Economics, 29(2), 203–214. Mausbach, B. T., Moore, R., Roesch, S., Cardenas, V., & Patterson, T. L. (2010). The relationship between homework compliance and therapy outcomes: An updated metaanalysis. Cognitive Therapy and Research, 34(5), 429–438. McNemar, Q. (1962). Psychological statistics (3rd ed.). New York: Wiley. Nunnally, J. C. (1967). Psychometric theory. McGraw-Hill series in psychology. New York: McGraw-Hill. R Core Team. (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing [Computer software]. Vienna, Austria. Retrieved from http://www.R-project.org/ S´anchez-Meca, J., & Mar´ın-Mart´ınez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and type i error. Quality & Quantity, 31(4), 385–399. Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology, 60(1), 29–60.
Original manuscript received January 30, 2013 Final version accepted October 4, 2013
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
An Empirical Investigation of Partial Effect Sizes in MetaAnalysis of Correlational Data Ariel M. Aloe
a
a
University of Northern Iowa Published online: 20 Dec 2013.
To cite this article: Ariel M. Aloe (2014) An Empirical Investigation of Partial Effect Sizes in Meta-Analysis of Correlational Data, The Journal of General Psychology, 141:1, 47-64, DOI: 10.1080/00221309.2013.853021 To link to this article: http://dx.doi.org/10.1080/00221309.2013.853021
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [York University Libraries] at 14:56 15 November 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
The Journal of General Psychology, 2014, 141(1), 47–64 C 2014 Taylor & Francis Group, LLC Copyright
Downloaded by [York University Libraries] at 14:56 15 November 2014
An Empirical Investigation of Partial Effect Sizes in Meta-Analysis of Correlational Data ARIEL M. ALOE University of Northern Iowa
ABSTRACT. The partial correlation and the semi-partial correlation can be seen as measures of partial effect sizes for the correlational family. Thus, both indices have been used in the meta-analysis literature to represent the relationship between an outcome and a predictor of interest, controlling for the effect of other variables in the model. This article evaluates the accuracy of synthesizing these two indices under different situations. Both partial correlation and the semi-partial correlation appear to behave as expected with respect to bias and root mean squared error (RMSE). However, the partial correlation seems to outperform the semi-partial correlation regarding Type I error of the homogeneity test (Q statistic). Although further investigation is needed to fully understand the impact of meta-analyzing partial effect sizes, the current study demonstrates the accuracy of both indices. Keywords: meta-analysis, partial effect sizes, regression results
META-ANALYSIS IS “THE STATISTICAL ANALYSIS of a large collection of analysis results from individual studies for the purposes of integrating findings” (Glass, 1976, p. 3). In meta-analysis, study results are transformed to a common metric and represented as effect sizes for analysis. Effect sizes are scale-free indices that assess the magnitude and direction of the relationship between variables. Nearly all effect-size indices, used in meta-analysis, have variances that can be computed from sufficient statistics representing the uncertainty in the effect-size values. Thus, effect sizes are a crucial component of any meta-analysis. Partial effect sizes for the r-family (correlation coefficient) of effects have been formally discussed by Becker and Wu (2007) and Aloe and Becker (2012). However, none of these sources empirically investigate the accuracy of the estimations nor the behavior of the homogeneity test when meta-analyzing partial effect sizes. The two partial effect sizes considered in this article are the partial correlation and the semi-partial correlation. Along with Keef and Roberts (2004), Address correspondence to Ariel M. Aloe, University of Northern Iowa, 613 Schindler Education Center, Cedar Falls, IA 50614, USA; [email protected] (e-mail). 47
Downloaded by [York University Libraries] at 14:56 15 November 2014
48
The Journal of General Psychology
who discussed partial effects in the context of standardized mean differences, we also refer to zero-order and partial effects to distinguish between the correlation coefficient (zero-order effects) and partial and semi-partial correlations (partial effects). There is extensive literature discussing the meta-analysis of zero order correlations (e.g., Field, 2001; Hafdahl, 2009; Hittner & May, 2012; Hunter & Schmidt, 1990). Nevertheless, the literature discussing the synthesis of partial effect sizes for the r-family of effects is, at best, very limited. A possible explanation is that many researchers, when conducting meta-analysis, treat partial effect sizes as zero order correlations (e.g., Bowman, 2010; Lynn & McCall, 2000; Mausbach, Moore, Roesch, Cardenas, & Patterson, 2010). Another possibility is that metaanalysts assume the statistical properties of zero order effects, such as the bivariate correlation, will transfer directly to partial effect sizes. Almost two decades ago, researchers argued that results from regression models can be treated as effect size indices (e.g., Becker & Schram, 1994; Hedges, Laine, & Greenwald, 1994). In order to confidently synthesize partial effect sizes, a better understanding of the general statistical properties of the synthesis of such partial effects is needed. As of today, there is no article that examines the behavior of partial effect sizes for the r-family. Thus, in this article we seek to evaluate the accuracy of the synthesis of partial effect sizes under fixed and random effect models. In addition, we study the behavior of the homogeneity test when partial or semi-partial correlations are synthesized. The overall organization of this article is as follows. First, the distinction between bivariate and partial effect sizes is discussed. The next section describes the partial and semi- partial correlation with their respective variances. Then, a brief overview of meta-analytical models is presented. A simulation study is presented and the article finishes with a set of general conclusions and recommendations for the meta-analyst. Bivariate and Partial Effects Bivariate effect sizes represent the bivariate association between a predictor (or independent) variable, and an outcome (or dependent) variable (for example, the Pearson product- moment correlation coefficient, r). In contrast, partial effect sizes (i.e., regression coefficients, partial correlations, and semi-partial correlations) represent the relationship between a predictor variable and an outcome variable while controlling for the effects of one or more additional variables. When predictors are correlated, it can have a significant impact on both the magnitude and direction of partial effects. The presence of collinearity can inflate correlation estimates, and in some cases can even reverse the sign of the bivariate correlation and its counterpart regression coefficient, partial correlation, and semi-partial correlation (Hocking, 1983). Additionally, when the relationship between the focal predictor and the outcome variable increases as a function of
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
49
another predictor being added to the model, suppression may occur. The presence of suppression may increase or decrease the magnitude of partial effect sizes, and in some cases can also reverse the sign with respect the bivariate correlation. Therefore, bivariate effects and partial effects represent different parameters and thus should not be treated as the same, should not be expected to have exactly the same properties as zero order effects, nor should they be combined within the same data set with zero order effects. Lastly, common meta-analytical techniques (e.g., Cooper, Hedges, & Valentine, 2009) can be used to synthesize partial effect sizes. However, one must take into account that each partial effect size could potentially arise from a different regression model (i.e., different sets of variables may be controlled or partialed out in different primary studies). When this happens, each partial effect in a metaanalysis could be estimating a different population parameter. This suggests that in some circumstances the synthesis of partial correlations or semi-partial correlations may not be possible or wise. In other circumstances modeling partial correlations may be more informative than reporting an overall partial effect. Accordingly, a meta-analyst faces two possible situations. In the first case, all the studies included in a meta-analysis of partial effect sizes may control for the same variables. This is not very likely in practice. Alternatively, the studies included in a meta-analysis of partial effect sizes may control for different covariates, which is more likely to be the case in practice. This suggests that when analyzing a collection of partial-effect values, the reviewer needs to include predictor variables that reflect differences in model complexity and are likely to influence the sizes of the partial correlation. Notation and Model The population regression model with p predictors can be written as Yj = β 0 + β 1 Xj1 + β 2 Xj2 + · · · + β p Xjp + ej , where β 0 is the intercept, β p is the regression coefficient(slope) associated with the variable Xp , and ej is the error term that follows a normal distribution such that ej ∼ N(0, σ 2). For consistency, we will refer to the variable of interest, our focal predictor, as Xf . For the case with two predictors, the regression coefficient can be represented as a function of the bivariate correlations such as (ryf − ry2 rf2 )/ (1 − r2f2 ) where ryf , ry2 , rf2 are the bivariate correlations between the focal predictor and the outcome, between the other predictor in the model and the outcome, and between the two predictors, respectively. Next, we discuss the partial and semi-partial correlations. The Semi-Partial Correlation The semi-partial correlation (e.g., Nunnally, 1967, p. 155) or part correlation (e.g., McNemar, 1962, pp. 167–168) is the correlation of a variable, say X1 , with the outcome, Y, after the portion of Y that can be linearly predicted from
Downloaded by [York University Libraries] at 14:56 15 November 2014
50
The Journal of General Psychology
another variable, say X 2 , has been removed from Y. For instance, suppose we are studying how well teacher self-efficacy predicts teacher performance. We know that years of experience is an important factor determining both teacher selfefficacy and teacher performance. But we are only interested in the relationship between teacher performance and teacher self-efficacy uninfluenced by years of experience. Thus, we want to partial out years of experience from the correlation, while simultaneously not ruling out years of experience. Suppose we want years of experience partialed out of the teacher self-efficacy scores but not the teacher performance scores. Then years of experience can be partialed only from the teacher self-efficacy score by the semi-partial correlation. This definition can be clarified by its computational formula for a model with only two predictors. Let ry1 , ry2 , and r12 be the bivariate correlations between the outcome and covariate one, the outcome and covariate two, and between covariates one and two, respectively. Thus, the semi-partial correlation between the outcome and covariate one is ry1 − ry2 r12 rsp1 = 2 1 − r12
(1)
Given that typically the semi-partial correlation can be obtained from multiple regression results, Aloe and Becker (2012) proposed the use of the semi-partial correlation as a partial effect size in meta-analysis. Specifically, the semi-partial correlation can be computed as
rspf
tf 1 − RY2 =√ (n − p − 1)
(2)
where tf is the t test of the regression coefficient β f in the multiple regression model, is the total variance accounted for by the full model with p number of predictors, n − p − 1 are the degrees of freedom, and n is the number of cases in the analysis. The variance vi (rsp ) can be estimated as (Aloe & Becker, 2012) vi (rsp ) =
RY4 − 2RY2 + RY2 (f ) + 1 − RY4 (f ) n
(3)
where R2y(f) = R2y – r2sp . The Partial Correlation Now suppose that we are interested only in the relationship between teacher performance and teacher self-efficacy, uninfluenced by years of experience. This is to say, we want to partial out years of experience from the correlation. If the correlation between teacher self-efficacy and teacher performance was obtained
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
51
from a sample with all subjects having the same number of years of experience, we would have controlled for years of experience in our design. The partial correlation achieves this statistically (Kerlinger & Pedhazur, 1973, p. 84). The partial correlation is the correlation between teacher self-efficacy and teacher performance in a group in which years of experience is constant. If all the members of the group had the same number of years of experience, the correlation of teacher self-efficacy and teacher performance would be equivalent to the partial correlation. Introductory statistics books (Cohen & Cohen, 1975; Nunnally, 1967) define the partial correlation for the case with two predictors and one outcome as ry1 − ry2 r12 rp1 = 2 2 1 − ry2 1 − r12
(4)
The partial correlation can also be obtained for models with more than two predictors as tf rp1 = 2 tf + (n − p − 1)
(5)
where tf is the t test of the regression coefficient β f in the multiple regression model, n − p − 1 are the degrees of freedom, p is the number of predictors, and n is the number of cases in the analysis. The variance vi (rp ) can be estimated as (Anderson,1984) 2 1 − rp2 vi rp = n−p−1
(6)
Brief Overview of Meta-Analytical Models Let Ti . . . Tk be the estimates of partial effect sizes (i.e., partial or semipartial correlations) from k independent studies, each Ti is an estimate of θ i , the true partial effect size. Typically, ei is the estimate error (i.e., the amount that Ti deviates from θ i ), which is assumed to be normally distributed with a mean of zero and variance vi . The random-effects model acknowledges two sources of variation: within-study sampling error (vi ) and between-studies variability, denoted by τ 2. The random-effects model can be represented as Ti = μ + ui + ei ,
(7)
and the variance of Ti is equal to vi + τ 2. Thus, the differences between the estimated partial effect sizes and the overall mean, μ, includes a component of
Downloaded by [York University Libraries] at 14:56 15 November 2014
52
The Journal of General Psychology
sampling variation for each study (ei = Ti − θ i ) as well as a term for the differences between the population effect size (ui = θ i − μ). In the absence of betweenstudies heterogeneity, τ 2 = 0, the random-effects model reduces to the fixedeffectmodel, having one source of variation (within-study sampling error). For the reader interested in a more detailed discussion, see Borenstein and colleagues (2010) and Hedges and Vevea (1998). It is standard to test whether the effect-size estimates are homogeneous—i.e., if they are estimating a common population partial effect size, θ . Specifically, to test the hypothesis H 0 : θ i = θ , i,. .., k, we use (Hedges, 1983) QT =
k (Ti − T¯ )2 vi i=1
(8)
where T¯ is the inverse-variance weighted mean partial effect size and vi is the within-study sampling variance for the partial effect size from study i, which will differ according to the index use. If QT exceeds the critical value of the χ 2 distribution with k − 1 degrees of freedom, the null hypothesis is rejected (Hedges & Olkin, 1985). Thus, a statistically significant QT indicates that the partial effectsize estimates from our k studies do not share the same population effect size. In such cases, potentially explainable variation exists among the partial effect sizes. Simulation To study the empirical performance of partial correlations and semi-partial correlations as measures of partial effect sizes, a simulation study was performed. The simulation was programmed in R (R Core Team, 2012). The Cholesky decomposition was used to obtain the desired correlations among variables. For simulating each individual study’s rp and rsp values, multiple regression models with two continuous predictors and a continuous outcome were considered. The number of studies (k) per meta-analysis was 10, 20, 30, and 40. For each condition, 5,000 replications were performed. For each replication rp and rsp , as well as their variances, were estimated. To simulate data for each pseudo study, scores from a multivariate normal distribution were generated, with means μi = 0.0, and variances vi = 1. By manipulating the covariances σ ij , different levels of collinearity and relatedness of the predictors to the outcome were obtained. A random sample of size n, with an average of 200, was generated for each multivariate normal distribution. The correlations between the predictors and the outcome took on the values .2, .4, and .6, and the correlation between the predictors was manipulated through the correlation matrix such as ρ ij = −.2, .2, .4, and .8, depending on the condition. Specifically, data were generated so that for each set of k studies, an individual study’s partial effect size arose from the same values of correlations. First, a correlation matrix with three variables (i.e., three unique correlation
Aloe
53
coefficients) was created for each study with a population matrix as ⎞ 1 ρyf ρy2 ⎝ ρfy 1 ρf 2 ⎠ , ρ2y ρ2f 1
Downloaded by [York University Libraries] at 14:56 15 November 2014
⎛
Next, a linear model was created for each study as Yj = β 0 + β 1 Xj1 + β 2 Xj2 + ej , where β 1 = β f , which is the slope for the focal predictor. For each study, the t-test associated with the focal predictor was extracted and the partial effect size and its variance were estimated using the corresponding formulas. In addition, to generate a random-effects model, random variability was added into the correlation between the outcome and the focal predictor (i.e., ρ yf ). Five values of between-study variability were selected (τ 2 = .0025, .005, .01, .05, and .10). Results Table 1 reports bias for the weighted average partial and semi-partial correlations for different values of between-study variance (τ 2), different values of number of studies (k), and displays the true partial effect size values (ρ pf and ρ spf ). Results suggest that the weighted average partial correlation and semi-partial correlating are remarkably accurate for all the simulated conditions even for the smallest number of studies (i.e., k = 10). The bias of the weighted average partial effects appeared to be consistently larger for the largest value of population heterogeneity (τ 2 = .10) compared to the other conditions. In addition, for the majority of the τ 2 = .10 conditions, results suggest an underestimation of the true partial effects for both partial and semi-partial correlations. The results also suggest that even in the presence of cooperative suppression and collinearity, the accuracy of the results is remarkable. If we let ryf represent the correlation between our outcome (Y) and our focal predictor (Xf ), cooperative suppression (Cohen & Cohen, 1975, p. 90) (i.e., reciprocal suppression Conger, 1974) occurs when ρ f 2 < 0. For example, for τ 2 = .00 (fixed-effect model), k = 10, ρ yf = .4, ρ y2 = .2, and ρ f2 = −.2, the bias for the weighted average partial and semi-partial correlations are −.0002 and .0038, respectively. It also appears that when there is relatively large collinearity between the predictors (i.e., ρ f2 = .80) the point estimate has excellent accuracy. For example, the result under τ 2 = .00 and k = 10 for ρ yf = .6, ρ y2 = .4, and ρ f2 = .8 indicates that the bias is .0043 for the partial correlation and a bias of −0.0012 for the semi-partial correlation. Table 2 gives root mean square error (RMSE) for the weighted average partial and semi- partial correlations for different values of between-study variance (τ 2) and different values of number of studies (k). The results also suggest that the RMSE of the weighted average partial and semi-partial correlations consistently increases as values of population heterogeneity (τ 2) increase. However, the RMSE
.286
.044
.40 −.20
.40
.40
.163
.20
.449
.367
.20 −.20
.20
.80 −.200
.245
.20 −.20
ρ spf
.20
ρ f2
ρ y2
ρ yf
k
.250 10 20 30 40 .167 10 20 30 40 .312 10 20 30 40 .048 10 20 30 40 −.218 10 20 30 40 .458 10 20 30 40 .375 10 20 30 40
ρ pf
.0001 .0005 .0005 .0006 .0003 .0003 .0001 .0000 −.0008 −.0006 −.0007 −.0003 −.0004 −.0006 .0002 .0001 .0004 .0003 .0004 .0004 −.0002 .0008 .0006 .0007 −.0001 .0011 .0009 .0007
r¯ spf .0027 .0034 .0035 .0036 .0021 .0023 .0022 .0022 .0031 .0040 .0040 .0045 .0002 .0001 .0011 .0009 .0026 .0031 .0031 .0032 .0038 .0053 .0053 .0055 .0034 .0052 .0050 .0049
r¯ pf
τ 2 = .00
.0012 .0008 .0007 .0010 .0007 .0011 .0004 .0004 −.0003 .0001 .0005 .0004 .0003 .0004 .0002 .0000 −.0010 −.0003 .0000 −.0006 .0016 .0024 .0022 .0017 .0023 .0014 .0017 .0012
r¯ spf .0040 .0038 .0038 .0040 .0027 .0032 .0026 .0025 .0037 .0044 .0048 .0048 .0010 .0012 .0010 .0007 .0039 .0033 .0030 .0037 .0056 .0065 .0064 .0059 .0061 .0053 .0056 .0052
r¯ pf
τ 2 = .0025
TABLE 1. Bias for Partial and Semi-Partial Correlations
.0010 .0007 .0010 .0008 .0005 .0002 .0007 .0004 .0002 .0002 −.0002 .0003 .0003 −.0002 .0002 .0002 −.0002 −.0003 −.0003 −.0005 .0013 .0014 .0019 .0016 .0010 .0009 .0012 .0013
r¯ spf .0037 .0037 .0041 .0039 .0024 .0023 .0028 .0026 .0043 .0047 .0045 .0051 .0010 .0006 .0010 .0010 .0032 .0035 .0036 .0037 .0053 .0058 .0063 .0060 .0047 .0049 .0053 .0055
r¯ pf
τ 2 = .005 r¯ pf
r¯ spf
r¯ pf
τ 2 = .05 r¯ spf
r¯ pf
(Continued on next page)
.0006 .0020 .0028 .0012 .0015 −.0008 .0014 .0002 −.0024 .0011 −.0010 −.0003 .0025 −.0009 .0008 .0009 −.0163 −.0183 −.0180 −.0186 −.0032 −.0023 −.0030 −.0031 −.0002 −.0015 −.0008 .0000
τ 2 = .10
.0008 .0035 .0004 .0025 −.0009 .0007 .0036 .0015 .0038 .0004 .0005 .0034 .0011 .0034 .0012 .0006 .0035 .0016 .0038 −.0004 .0016 .0035 .0015 .0030 .0004 .0007 .0028 −.0013 .0003 −.0020 .0009 .0029 .0000 .0016 .0002 .0003 .0024 .0009 .0026 −.0009 .0001 .0039 .0020 .0051 −.0038 .0012 .0054 −.0004 .0025 −.0007 .0000 .0040 .0009 .0040 −.0026 .0005 .0046 .0008 .0039 −.0019 −.0016 −.0011 −.0011 −.0006 .0018 .0008 .0016 −.0001 .0004 −.0012 −.0002 .0005 −.0001 .0005 .0003 −.0001 .0006 .0001 .0006 .0003 −.0013 .0039 .0030 −.0021 .0142 .0000 .0027 .0019 −.0008 .0162 −.0005 .0033 .0025 −.0015 .0158 .0001 .0026 .0033 −.0024 .0163 .0021 .0058 .0012 .0040 −.0050 .0020 .0060 .0025 .0054 −.0042 .0017 .0056 .0006 .0035 −.0049 .0017 .0057 .0020 .0049 −.0049 .0020 .0055 .0007 .0034 −.0023 .0013 .0050 .0015 .0043 −.0034 .0013 .0050 .0014 .0041 −.0028 .0012 .0050 .0009 .0037 −.0020
r¯ spf
τ 2 = .01
Downloaded by [York University Libraries] at 14:56 15 November 2014
54 The Journal of General Psychology
.60
ρ yf
.40
.694
.480
.467
.40
.80
.572
.20
−.20
.653
.133
.80
−.20
.262
.40
.20
.490
−.20
.40
ρ spf
ρ f2
ρ y2
.509
.524
.757
.583
.667
.145
.286
.535
ρ pf
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
k
−.0012 −.0012 −.0008 −.0008 −.0005 −.0005 −.0004 −.0005 −.0001 −.0002 −.0002 .0000 −.0002 .0005 .0005 .0005 .0003 .0008 .0005 .0008 −.0010 −.0011 −.0012 −.0009 −.0011 −.0011 −.0008 −.0009 −.0012 −.0012 −.0010 −.0008
r¯ spf .0045 .0052 .0057 .0059 .0033 .0037 .0040 .0040 .0020 .0020 .0022 .0025 .0042 .0053 .0056 .0057 .0047 .0056 .0056 .0060 .0040 .0045 .0046 .0050 .0044 .0052 .0057 .0058 .0043 .0049 .0054 .0058
r¯ pf
τ 2 = .00
.0010 .0008 .0008 .0005 −.0003 .0003 .0004 .0005 .0002 .0004 .0010 .0002 .0033 .0032 .0033 .0035 .0018 .0019 .0028 .0023 .0017 .0020 .0014 .0019 .0009 .0008 .0007 .0009 .0008 .0019 .0012 .0008
r¯ spf .0062 .0062 .0063 .0060 .0034 .0043 .0045 .0046 .0022 .0025 .0031 .0022 .0064 .0062 .0063 .0066 .0055 .0057 .0066 .0061 .0051 .0052 .0047 .0053 .0060 .0060 .0061 .0064 .0055 .0068 .0061 .0056
r¯ pf
τ 2 = .0025
.0003 .0008 .0001 .0002 .0003 .0001 .0000 −.0003 −.0004 .0005 −.0001 −.0003 .0019 .0028 .0027 .0029 .0014 .0024 .0024 .0028 .0013 .0014 .0015 .0016 .0003 .0004 .0003 .0006 .0014 .0008 .0010 .0009
r¯ spf .0057 .0067 .0060 .0061 .0043 .0043 .0043 .0040 .0015 .0027 .0021 .0019 .0053 .0061 .0060 .0062 .0053 .0065 .0064 .0069 .0050 .0052 .0052 .0055 .0057 .0062 .0061 .0065 .0064 .0061 .0063 .0062
r¯ pf
τ 2 = .005
TABLE 1. Bias for Partial and Semi-Partial Correlations (Continued)
.0012 .0014 .0013 .0011 .0008 −.0001 .0006 .0005 .0020 .0002 .0008 .0002 .0034 .0029 .0034 .0032 .0022 .0018 .0023 .0028 .0019 .0018 .0016 .0017 .0009 .0009 .0009 .0015 .0001 .0013 .0007 .0009
r¯ spf .0059 .0064 .0063 .0062 .0044 .0037 .0044 .0043 .0040 .0020 .0028 .0021 .0060 .0056 .0061 .0059 .0057 .0053 .0058 .0063 .0046 .0045 .0044 .0045 .0057 .0057 .0059 .0065 .0042 .0055 .0049 .0051
r¯ pf
τ 2 = .01
−.0025 −.0021 −.0011 −.0007 .0001 −.0008 .0005 .0004 −.0053 −.0019 −.0010 −.0017 −.0064 −.0040 −.0049 −.0040 .0005 .0000 −.0009 −.0010 −.0161 −.0152 −.0160 −.0150 −.0025 −.0023 −.0020 −.0028 −.0203 −.0190 −.0210 −.0221
r¯ spf .0003 .0009 .0020 .0025 .0027 .0018 .0032 .0032 −.0045 −.0008 .0001 −.0006 −.0049 −.0024 −.0033 −.0023 .0029 .0025 .0017 .0015 −.0178 −.0166 −.0174 −.0164 .0005 .0008 .0013 .0004 −.0179 −.0165 −.0187 −.0199
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
−.0101 −.0115 −.0112 −.0104 −.0024 −.0035 −.0034 −.0023 −.0148 −.0119 −.0117 −.0123 −.0247 −.0226 −.0221 −.0227 −.0117 −.0131 −.0129 −.0138 −.0404 −.0401 −.0404 −.0400 −.0139 −.0167 −.0172 −.0155 −.0612 −.0586 −.0605 −.0608
r¯ spf −.0098 −.0115 −.0110 −.0102 −.0007 −.0018 −.0017 −.0006 −.0141 −.0109 −.0106 −.0113 −.0242 −.0222 −.0216 −.0222 −.0099 −.0111 −.0111 −.0119 −.0472 −.0467 −.0470 −.0466 −.0124 −.0154 −.0160 −.0141 −.0603 −.0577 −.0598 −.0601
r¯ pf
τ 2 = .10
Aloe 55
.40
.20
.20
.20
.40
ρ y2
ρ yf
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
−.20
.20
−.20
.80
.40
−.20
.20
k
ρ f2
.0218 .0151 .0121 .0107 .0225 .0156 .0125 .0109 .0198 .0140 .0113 .0098 .0216 .0149 .0121 .0106 .0208 .0144 .0117 .0099 .0184 .0128 .0104 .0090 .0203 .0141 .0113 .0099
r¯ spf
.0225 .0158 .0129 .0116 .0232 .0163 .0131 .0114 .0216 .0155 .0128 .0115 .0238 .0165 .0134 .0118 .0229 .0160 .0131 .0113 .0190 .0139 .0118 .0106 .0210 .0152 .0125 .0112
r¯ pf
τ 2 = .00
.0331 .0227 .0181 .0160 .0329 .0225 .0187 .0161 .0309 .0213 .0176 .0149 .0340 .0233 .0188 .0163 .0445 .0303 .0254 .0214 .0303 .0206 .0172 .0145 .0315 .0215 .0178 .0150
r¯ spf .0342 .0237 .0190 .0169 .0340 .0234 .0195 .0168 .0340 .0237 .0198 .0170 .0376 .0258 .0209 .0180 .0490 .0335 .0281 .0239 .0313 .0218 .0185 .0158 .0326 .0225 .0189 .0162
r¯ pf
τ 2 = .0025
TABLE 2. RMSE for Partial and Semi-Partial Correlations
.0278 .0190 .0156 .0133 .0283 .0193 .0160 .0136 .0264 .0180 .0144 .0127 .0278 .0196 .0158 .0135 .0349 .0242 .0196 .0164 .0253 .0174 .0143 .0121 .0264 .0182 .0148 .0125
r¯ spf .0288 .0199 .0166 .0142 .0292 .0200 .0167 .0143 .0291 .0202 .0164 .0148 .0308 .0217 .0175 .0150 .0385 .0269 .0218 .0184 .0262 .0185 .0157 .0135 .0273 .0192 .0160 .0138
r¯ pf
τ 2 = .005
.0404 .0281 .0230 .0196 .0409 .0280 .0223 .0196 .0391 .0266 .0220 .0190 .0421 .0287 .0238 .0205 .0594 .0410 .0329 .0284 .0392 .0270 .0216 .0187 .0396 .0277 .0217 .0193
r¯ spf .0417 .0291 .0239 .0205 .0423 .0290 .0231 .0203 .0429 .0297 .0244 .0213 .0465 .0318 .0264 .0227 .0654 .0452 .0364 .0314 .0403 .0281 .0227 .0198 .0408 .0287 .0227 .0203
r¯ pf
τ 2 = .01
.0788 .0545 .0433 .0381 .0796 .0561 .0439 .0383 .0788 .0540 .0445 .0373 .0835 .0593 .0467 .0400 .1237 .0846 .0691 .0595 .0783 .0539 .0422 .0377 .0791 .0546 .0432 .0379
r¯ spf .1077 .0743 .0608 .0522 .1091 .0759 .0614 .0524 .1071 .0736 .0589 .0519 .1150 .0797 .0650 .0564 .1646 .1116 .0917 .0785 .1059 .0728 .0589 .0498 .1084 .0734 .0599 .0514
r¯ spf .1104 .0762 .0623 .0536 .1120 .0779 .0630 .0538 .1170 .0805 .0643 .0567 .1263 .0876 .0714 .0620 .1799 .1220 .1003 .0859 .1078 .0742 .0599 .0507 .1109 .0750 .0613 .0525
r¯ pf
τ 2 = .10
(Continued on next page)
.0809 .0561 .0446 .0393 .0819 .0577 .0452 .0395 .0865 .0593 .0489 .0410 .0921 .0653 .0514 .0441 .1355 .0927 .0756 .0651 .0798 .0552 .0431 .0387 .0810 .0560 .0443 .0389
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
56 The Journal of General Psychology
.60
ρ yf
.40
.20
10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40
−.20
.40
.80
.40
−.20
.20
−.20
.80
.40
k
ρ f2
ρ y2
.0170 .0118 .0093 .0082 .0206 .0138 .0111 .0097 .0214 .0148 .0119 .0102 .0135 .0094 .0076 .0065 .0158 .0109 .0087 .0078 .0123 .0086 .0069 .0059 .0171 .0119 .0096 .0084 .0173 .0119 .0096 .0081
r¯ spf
.0177 .0130 .0110 .0102 .0225 .0155 .0127 .0113 .0236 .0164 .0133 .0115 .0138 .0106 .0093 .0085 .0164 .0122 .0104 .0098 .0108 .0082 .0073 .0069 .0179 .0132 .0112 .0103 .0183 .0131 .0111 .0102
r¯ pf
τ 2 = .00
.0297 .0202 .0164 .0140 .0324 .0223 .0183 .0157 .0448 .0308 .0250 .0216 .0284 .0193 .0159 .0140 .0290 .0198 .0162 .0142 .0271 .0190 .0152 .0131 .0313 .0218 .0172 .0147 .0427 .0296 .0237 .0204
r¯ spf .0322 .0223 .0186 .0160 .0357 .0249 .0206 .0179 .0495 .0341 .0277 .0240 .0288 .0199 .0167 .0150 .0296 .0206 .0174 .0154 .0281 .0200 .0162 .0142 .0341 .0241 .0194 .0169 .0465 .0326 .0263 .0228
r¯ pf
τ 2 = .0025
.0240 .0164 .0134 .0114 .0275 .0185 .0154 .0129 .0349 .0240 .0196 .0169 .0220 .0152 .0126 .0111 .0235 .0161 .0134 .0118 .0214 .0146 .0119 .0102 .0251 .0174 .0142 .0119 .0322 .0224 .0181 .0159
r¯ spf .0260 .0184 .0153 .0135 .0304 .0207 .0173 .0147 .0386 .0267 .0218 .0188 .0224 .0160 .0136 .0123 .0241 .0172 .0147 .0134 .0217 .0154 .0127 .0114 .0271 .0194 .0162 .0141 .0351 .0247 .0204 .0182
r¯ pf
τ 2 = .005
TABLE 2. RMSE for Partial and Semi-Partial Correlations (Continued)
.0380 .0262 .0214 .0184 .0419 .0288 .0228 .0202 .0587 .0404 .0338 .0292 .0369 .0253 .0210 .0181 .0380 .0259 .0209 .0186 .0361 .0248 .0201 .0173 .0399 .0277 .0223 .0195 .0579 .0394 .0325 .0279
r¯ spf .0412 .0288 .0238 .0206 .0462 .0318 .0254 .0226 .0649 .0446 .0374 .0323 .0372 .0258 .0217 .0188 .0387 .0266 .0217 .0196 .0380 .0261 .0214 .0187 .0434 .0303 .0247 .0219 .0631 .0431 .0357 .0306
r¯ pf
τ 2 = .01
.0766 .0522 .0424 .0362 .0832 .0580 .0462 .0398 .1256 .0849 .0683 .0584 .0731 .0505 .0417 .0349 .0761 .0514 .0420 .0358 .0693 .0492 .0414 .0357 .0804 .0542 .0447 .0390 .1162 .0796 .0661 .0589
r¯ spf .0830 .0565 .0459 .0393 .0914 .0637 .0508 .0438 .1380 .0932 .0750 .0642 .0737 .0508 .0419 .0352 .0772 .0523 .0427 .0363 .0736 .0525 .0441 .0382 .0875 .0589 .0486 .0424 .1270 .0865 .0714 .0633
r¯ pf
τ 2 = .05
Downloaded by [York University Libraries] at 14:56 15 November 2014
.1018 .0713 .0577 .0496 .1141 .0801 .0631 .0536 .1588 .1118 .0911 .0791 .0983 .0699 .0586 .0509 .1007 .0704 .0572 .0492 .0961 .0734 .0634 .0581 .1061 .0767 .0618 .0536 .1585 .1144 .1014 .0928
r¯ spf .1101 .0770 .0622 .0534 .1251 .0877 .0690 .0587 .1744 .1225 .0997 .0866 .0993 .0706 .0590 .0512 .1022 .0713 .0578 .0496 .1037 .0803 .0701 .0646 .1155 .0832 .0667 .0577 .1724 .1229 .1080 .0980
r¯ pf
τ 2 = .10
Aloe 57
58
The Journal of General Psychology
TABLE 3. Type I Error for Partial and Semi-Partial Correlations
Downloaded by [York University Libraries] at 14:56 15 November 2014
k = 10
k = 20
k = 30
k = 40
ρ yf
ρ y2
ρ f2
r¯ spf
r¯ pf
r¯ spf
r¯ pf
r¯ spf
r¯ pf
r¯ spf
r¯ pf
.20
.20
−.20 .20 −.20 .40 .80 −.20 .20 −.20 .40 .80 −.20 .20 −.20 .40 .80
.054 .058 .090 .131 .102 .022 .034 .038 .090 .118 .002 .006 .006 .038 .041
.062 .058 .058 .066 .059 .061 .060 .055 .059 .064 .057 .053 .058 .056 .057
.057 .072 .115 .172 .138 .016 .034 .035 .120 .162 .000 .003 .003 .038 .038
.070 .070 .067 .066 .069 .061 .068 .066 .069 .069 .053 .060 .059 .065 .060
.054 .075 .139 .212 .166 .012 .021 .031 .138 .179 .000 .001 .001 .032 .037
.067 .072 .080 .076 .071 .067 .061 .067 .069 .066 .062 .066 .059 .063 .059
.054 .074 .156 .250 .189 .010 .022 .024 .152 .227 .000 .001 .000 .035 .038
.070 .072 .087 .076 .077 .068 .070 .064 .065 .065 .068 .066 .055 .075 .061
.40 .40
.20 .40
.60
.20 .40
values clearly decrease as k increases, which is not surprising because as k increases, we have more data with which to make inf ferences about the model and its quality. Again, the results suggests that these indices behave as expected even in presence of collinearity. For example, the result for τ 2 = .00, k = 10, ρ yf = .6, ρ y2 = .4, and ρ f2 = .8 indicates that the RMSEs are .0173 and .0183 for rspf and rpf , respectively. These values are comparable to other conditions holding τ 2 and k constant. However, the results suggest that RMSE is consistently larger for conditions with a larger amount of collinearity than conditions in which ρ f2 is less than .80. Table 3 displays empirical Type I error rates of the Q test for both partial effect sizes. The empirical rejection reported in Table 3 is an indication of how well the Q test detects heterogeneity. When τ 2 = .00, each independent rpf and rspf were generated from the same patterns of correlations with identical numbers of predictors. Given that 5,000 replications were performed, the 95% confidence interval for the proportions of Q values greater than their corresponding1 critical value was expected to be .05 ± 1.96(.05(1 − .05)/ 5000)1/2 = .05 ± .006 = [.044, .056]. However, for the semi-partial correlation, Type I error rates were significantly higher or significantly lower than their nominal value for all conditions considered. On the other hand, for the partial correlation the results suggest that the rejection rate is close to nominal for the majority of the conditions, but somewhat higher
Downloaded by [York University Libraries] at 14:56 15 November 2014
Aloe
59
than the nominal value. This aligns with the findings of Alexander, Scozzaro, and Borodkin(1989) who reported higher than nominal Type I error rates for test of homogeneity of bivariate correlations. In addition, studying the behavior of the Q test for bivariate correlations, Viechtbauer (2007) also noticed that when k increases, the probability of rejecting the null hypothesis increases as well. This finding is also true in the current study of the partial correlation as a measure of partial effect size. Currently, there are no other simulation studies reporting on Type I error rates for the semi-partial correlation or partial correlation. However, higher than nominal Type I error rates for test of homogeneity of bivariate correlations were previously reported (Alexander et al., 1989; S´anchez-Meca & Mar´ın-Mart´ınez, 1997; Viechtbauer, 2007). Yet, the lower than nominal Type I error rates are unexpected, suggesting that the results of the Q test, when synthesizing semi-partial correlations may not be that informative. The empirical rejection rates are reported in Figure 1 for the partial correlation and Figure 2 for the semi-partial correlation. Both represent the rejection rate as a function of τ 2 and k for different combinations of correlation values. Our findings suggest that increasing τ 2 and/or k results in a higher probability of detecting heterogeneity. Similar results also were reported in studies that use the bivariate correlationas an effect size measure (e.g., Viechtbauer, 2007). The results also indicate that in presence of collinearity (see Figure 1 and 2: m, n, and o) the null appears to be rejected much sooner even for small k values (i.e., k = 10) than for conditions with smaller ρ f2 values. Finally, the true magnitude of the partial correlation and semi-partial correlation appear to have an effect on the rejection rate. For instance, comparing in Figure 1 condition (f), ρ pf = .757, and condition (l), ρ pf = .524, the probability of detecting heterogeneity appears higher for all k values. Last, attention should be given to the true value of the partial effect sizes (column 4 and 5 in Table 1). It is well known that the true magnitude of the partial correlation and semi-partial correlation are affected by the values of the correlation matrix and the number of predictors in the model. However, under the simplest condition the bias and RMSE of the estimates is quite remarkable, making the partial correlation and semi-partial correlation promising partial effect sizes. Discussion and Practical Recommendations We have discussed the partial correlation and semi-partial correlation as a measure of partial effect size for the r-family. Overall, the results of demonstrate that the weighted average partial correlation and semi-partial correlation are accurate for all the simulated conditions, even for the smallest number of studies and collinearity between the predictors. In addition, results suggest that the rejection rate of the homogeneity test is close to, but somewhat higher than, the nominal for the majority of the conditions for the partial correlation. However, the use of the homogeneity test for the semi-partial correlation is questionable.
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
1.0 0.8 0.6 0.4 0.2
0
●
●
●
●
●
●
(a)
(d)
●
(g)
●
(j)
●
●
.0025 .005
●
.01
(m)
●
.05
●
ρY1 = .2,ρY2 = .4,ρ12 = .8
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
●
(b)
●
(e)
●
●
(h)
●
(k)
●
●
.01
(n)
●
.05
●
Amount of Heterogeneity
.0025 .005
●
ρY1 = .4,ρY2 = .4,ρ12 = .8
●
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .4,ρY2 = .2,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
(c)
●
●
●
(f)
●
●
(i)
●
●
●
(l)
●
●
.0025 .005
●
.01
(o)
●
.05
●
ρY1 = .6,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .6,ρY2 = .4,ρ12 = .4
●
●
ρY1 = .6,ρY2 = .2,ρ12 = −.2
●
ρY1 = .6,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .2,ρ12 = .2
Downloaded by [York University Libraries] at 14:56 15 November 2014
FIGURE 1. Rejection rate of the Q test for partial correlations.
RR
.10
●
●
●
●
●
k ●
40
30
20
10
60 The Journal of General Psychology
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
1.0 0.8 0.6 0.4 0.2 0.0
0
●
●
●
●
●
●
(a)
(d)
●
(g)
●
●
(j)
●
.0025 .005
●
●
.01
(m)
●
.05
●
ρY1 = .2,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .2,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .4,ρ12 = −.2
●
●
●
ρY1 = .2,ρY2 = .2,ρ12 = .2
.10
●
●
●
●
●
0
●
●
●
●
●
●
(b)
●
(e)
●
(h)
●
●
(k)
●
.01
(n)
●
.05
●
Amount of Heterogeneity
.0025 .005
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .4,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .4,ρY2 = .2,ρ12 = −.2
●
●
●
ρY1 = .4,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .4,ρY2 = .2,ρ12 = .2
FIGURE 2. Rejection rate of the Q test for semi-partial correlations.
RR
.10
●
●
●
●
●
0
●
●
●
●
●
(c)
●
●
(f)
●
●
(i)
●
●
(l)
●
●
.0025 .005
●
.01
(o)
●
.05
●
ρY1 = .6,ρY2 = .4,ρ12 = .8
●
●
ρY1 = .6,ρY2 = .4,ρ12 = .4
●
●
●
ρY1 = .6,ρY2 = .2,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .4,ρ12 = −.2
●
●
ρY1 = .6,ρY2 = .2,ρ12 = .2
Downloaded by [York University Libraries] at 14:56 15 November 2014
.10
●
●
●
●
●
k ●
40
30
20
10
Aloe 61
Downloaded by [York University Libraries] at 14:56 15 November 2014
62
The Journal of General Psychology
The overall results suggest that the meta-analysis of the partial and semipartial correlations appear to have desired properties regarding bias and accuracy. However, there are several limitations in the synthesis of partial effect size indices that meta-analysts should take into account. First, one must follow the inclusion criteria of meta-analysis. The studies included and the treatment of the data need to align with the inclusion criteria. If the inclusion criteria are set to synthesize bivariate associations, no measure of partial association should be included (i.e., semi-partial correlation or partial correlation). Thus, if the meta-analyst is planning to allow for measures of partial association to be included in their syntheses, this should be clearly stated in their inclusion criteria section and formulas used should be provided. It is noteworthy that having different sets of predictors leads to different interpretations of the partial and semi-partial correlation for each model. Thus, when interpreting overall results the meta-analyst should attempt to interpret the results while taking into account what has been partialed out. This can be extremely difficult since, in the social sciences, sometimes we find models that have more than 20 predictors (e.g., Chaney, 1995). A possible alternative is to take into account the larger constructs that the variables represent, not the specific predictors. For instance, in a meta-analysis of student academic achievement and teacher expectations, studies may control for different sets of variables, but all of the studies to some degree might control for students, teacher, and/or school characteristics. The decision about which partial effect size to use should be made by taking into account the research question, the interpretation that the meta-analyst wishes to achieve, and what information is reported in the primary studies. For instance, if a primary study failed to report the total variance explained (R2y ), the semipartial correlation could not be computed for this study. Lastly, partial effects can be larger or smaller than their corresponding bivariate correlations. Thus, it is recommended that meta-analysts do not treat partial effect sizes as bivariate effects. It is also advisable that meta-analyst do not combine, in the same data set, bivariate correlations with partial effects. AUTHOR NOTE Ariel M. Aloe is an assistant professor in the Department of Educational Psychology and Foundations, University of Northern Iowa. He specializes in research synthesis, meta-analysis, quantitative methods, and teacher quality. ACKNOWLEDGMENTS The author is grateful to Betsy Becker and Christopher Thompson for their feedback in an earlier version of this article.
Aloe
63
FUNDING This research has been partially supported by a grant from the National Science Foundation (NSF 1252263). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Downloaded by [York University Libraries] at 14:56 15 November 2014
REFERENCES Alexander, R. A., Scozzaro, M. J., & Borodkin, L. J. (1989). Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis. Psychological Bulletin, 106(2), 329–331. Aloe, A. M., & Becker, B. J. (2012). An effect size for regression predictors in metaanalysis. Journal of Educational and Behavioral Statistics, 37(2), 278–297. Anderson, T. W. (1984). An introduction to multivariate statistical analysis (2nd ed.). New York: Wiley. Becker, B. J., & Schram, C. M. (1994). Examining explanatory models through research synthesis. In The handbook of research synthesis (pp. 357–381). New York: Russell Sage Foundation. Becker, B. J., & Wu, M.-J. (2007). The synthesis of regression slopes in meta-analysis. Statistical Science, 22(3), 414–429. doi:10.1214/07-STS243 Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. Bowman, N. A. (2010). College diversity experiences and cognitive development: A metaanalysis. Review of Educational Research, 80(1), 4–33. Chaney, B. (1995). Student outcomes and the professional preparation of eight-grade teachers in science and mathematics: NSF/NELS. Rockville, MD: Westat. Cohen, J., & Cohen, P. (1975). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum. Conger, A. J. (1974). A revised definition for suppressor variables: A guide to their identification and interpretation. Educational & Psychological Measurement, 34(1), 35–46. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation. Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6(2), 161–180. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Research, 5(10), 3–8. Hafdahl, A. R. (2009). Improved fisher z estimators for univariate random-effects metaanalysis of correlations. British Journal of Mathematical and Statistical Psychology, 62(2), 233–261. Hedges, L. V. (1983). A random effects model for effect sizes. Psychological Bulletin, 93(2), 388–395. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504. Hedges, L. V., Laine, R. D., & Greenwald, R. (1994). An exchange: part I: Does money matter? A meta-analysis of studies of the effects of differential school inputs on student outcomes. Educational Researcher, 23(3), 5–14.
Downloaded by [York University Libraries] at 14:56 15 November 2014
64
The Journal of General Psychology
Hittner, J. B., & May, K. (2012). How accurate is the pearson r-from-z approximation: A Monte Carlo stimulation study. Journal of General Psychology, 139(2), 68–77. Hocking, R. R. (1983). Developments in linear regression methodology: 1959–1982. Technometrics, 25(3), 219–245. Retrieved from http://www.jstor.org/stable/1268603 Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage Publications, Inc. Keef, S. P., & Roberts, L. A. (2004). The meta-analysis of partial effect sizes. British Journal of Mathematical & Statistical Psychology, 57(1), 97–129. Kerlinger, F., & Pedhazur, E. (1973). Multiple regression in behavioral research. New York, NY: Holt, Rinehart, and Winston. Lynn, M., & McCall, M. (2000). Gratitude and gratuity: A meta-analysis of research on the service-tipping relationship. Journal of Socio-Economics, 29(2), 203–214. Mausbach, B. T., Moore, R., Roesch, S., Cardenas, V., & Patterson, T. L. (2010). The relationship between homework compliance and therapy outcomes: An updated metaanalysis. Cognitive Therapy and Research, 34(5), 429–438. McNemar, Q. (1962). Psychological statistics (3rd ed.). New York: Wiley. Nunnally, J. C. (1967). Psychometric theory. McGraw-Hill series in psychology. New York: McGraw-Hill. R Core Team. (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing [Computer software]. Vienna, Austria. Retrieved from http://www.R-project.org/ S´anchez-Meca, J., & Mar´ın-Mart´ınez, F. (1997). Homogeneity tests in meta-analysis: A Monte Carlo comparison of statistical power and type i error. Quality & Quantity, 31(4), 385–399. Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology, 60(1), 29–60.
Original manuscript received January 30, 2013 Final version accepted October 4, 2013
