This article was downloaded by: [Umeå University Library] On: 19 November 2014, At: 18:21 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study a
James B. Hittner & Kim May
a
a
College of Charleston Published online: 04 Apr 2012.
To cite this article: James B. Hittner & Kim May (2012) How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study, The Journal of General Psychology, 139:2, 68-77, DOI: 10.1080/00221309.2012.661376 To link to this article: http://dx.doi.org/10.1080/00221309.2012.661376
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [Umeå University Library] at 18:21 19 November 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
The Journal of General Psychology, 2012, 139(2), 68–77 C 2012 Taylor & Francis Group, LLC Copyright
Downloaded by [Umeå University Library] at 18:21 19 November 2014
How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study JAMES B. HITTNER KIM MAY College of Charleston
ABSTRACT. The Pearson r-from-Z approximation estimates the sample correlation (as an effect size measure) from the ratio of two quantities: the standard normal deviate equivalent (Z-score) corresponding to a one-tailed p-value divided by the square root of the total (pooled) sample size. The formula has utility in meta-analytic work when reports of research contain minimal statistical information. Although simple to implement, the accuracy of the Pearson r-from-Z approximation has not been empirically evaluated. To address this omission, we performed a series of Monte Carlo simulations. Results indicated that in some cases the formula did accurately estimate the sample correlation. However, when sample size was very small (N = 10) and effect sizes were small to small-moderate (ds of 0.1 and 0.3), the Pearson r-from-Z approximation was very inaccurate. Detailed figures that provide guidance as to when the Pearson r-from-Z formula will likely yield valid inferences are presented. Keywords: Pearson r-from-Z, correlation, meta-analysis, effect size, Monte Carlo simulation
WHEN PERFORMING A QUANTITATIVE RESEARCH SYNTHESIS, or metaanalysis, researchers rarely have access to raw data. Instead, the norm is to incorporate published summary statistics (e.g., t, F, χ 2, and odds ratio) into conversion formulas to compute standardized effect size estimates. Moreover, many of the common effect size measures, such as Cohen’s d (Cohen, 1988), Hedges’ g (Hedges, 1982), and the product-moment correlation, r, can be converted from one effect size metric to the other (see Cooper & Hedges, 1994, and Rosenthal & Rosnow, 1991, for relevant formulas). Although different researchers may prefer working with different effect size measures, Rosenthal (1991) argued that the product-moment correlation, r, has several desirable properties relative to other indices. First, unlike effect size measures based on mean differences in which Address correspondence to James B. Hittner, PhD, Department of Psychology, College of Charleston, 66 George Street, Charleston, SC 29424, USA; [email protected] (e-mail). 68
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Hittner & May
69
the sample size component varies based on whether observations are correlated or independent, the correlation coefficient, r, does not require any special adjustments. Second, relative to other effect size measures, the correlation arguably is more intuitively meaningful and can readily be interpreted in terms of practical utility. For example, the square of the correlation represents the amount of variance shared between two variables. In addition, researchers from various fields might wish to convert rs to binomial effect size displays (BESDs; Rosenthal, 1994) to demonstrate the real-world importance of an effect size indexed by a correlation coefficient. Although not without critics (Hsu, 2004), BESDs can be useful for gauging the improvement in success rate associated, for example, with initiating a new treatment procedure or incorporating a new predictor variable (Rosenthal, 1991; 1994). A third desirable property of r is its greater flexibility relative to other commonly used effect size measures, such as Cohen’s d. For example, r can always be computed from summary statistics whenever d can, but d cannot always be calculated whenever r can (see Rosenthal, 1991, p. 18). Although many published empirical research articles include the data necessary to compute correlation-based measures of effect size, some reports do not. In addition, research that is presented or summarized in other formats, such as conference presentations, poster handouts, published abstracts, press releases, and media reports often contain even less statistical information. These mediums for communicating research might emphasize sound bites and bullet points at the expense of statistical data. When data are presented, they often consist of p-values and group sample sizes only. In some reports, numerical data are absent altogether and the results merely are presented as either “statistically significant” or “non-significant.” For those research reports that do include specific p-values and group sample sizes, Rosenthal (1991) wrote in his popular meta-analysis book that researchers can estimate an effect size, r, by converting the obtained one-tailed p-value to its standard normal deviate equivalent (Z-score) and then computing r from the following formula, where N equals the total (pooled) sample size: Z r=√ N
(1)
Although Rosenthal’s simple conversion formula is convenient and easy to use, the formula’s accuracy has not, to our knowledge, been the subject of rigorous empirical evaluation. Delineating the conditions under which Rosenthal’s Pearson r-from-Z approximation accurately estimates the sample correlation will provide researchers with guidance as to when equation (1) will yield valid inferences. Given the absence of such empirically-driven guidance, the purpose of the present study was to examine, through a series of Monte Carlo simulations, the accuracy of equation (1) in estimating the sample correlation under a variety of sample size, distributional, and population effect size conditions.
70
The Journal of General Psychology
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Method The approach taken here required the creation of repeated sets of two independent samples with a specified skew and effect size. We note here that because the skew of the F-distribution is a direct function of the numerator and denominator degrees of freedom, it is relatively straightforward to specify an F-distribution with a particular value for skewness. This was accomplished by using SAS data step language in the following manner. First, two random variables, x1 and x2 , were generated from a uniform distribution (so that all values were equally likely). Then, a third variable, y1 , was created by transforming x1 to a quantile from an Fdistribution (i.e., a value of F) with the desired level of skewness. A fourth variable, y2 , was created by also transforming x2 to a quantile from an F-distribution with the specified skewness and then adding to y2 a multiple of the standard deviation of the given F-distribution, thereby adding the effect size in the form of a specified value of Cohen’s d. This procedure produced the two independent samples mentioned above. Aside from the effect size, each pair of y1 , y2 variables had the same level of skewness. In other words, we did not examine cases in which the two distributions have unequal skew. Then for each simulated data set (values of the simulated dependent variables, y1 and y2 ), a two independent-samples t-test on the means (of y1 vs. y2 ) was performed to generate a one-tailed p-value. From each one-tailed p, the corresponding standard normal deviate, or Z-score, was obtained and the estimated correlation from equation (1) was calculated. Then the actual observed correlation between the dependent variable (a single Y vector formed by concatenating y1 and y2 ) and the independent variable (a binary variable indexing group membership in y1 vs. y2 ) was computed. This is, of course, the observed effect size measure rIV,DV , which is estimated by equation (1). Finally, for each simulated sample, the numerical difference between the estimated correlation and the actual observed sample correlation was calculated and retained for analysis. Regarding the parameter specifications, we examined four different sample sizes (Ns = 10, 30, 60, and 100), four effect size conditions (ds = 1.0, 0.6, 0.3, and 0.1), and four different data distributions (skews = 0, 2, 5, and 10). The skew = 0 condition represents the normal distribution. The F distributions used to generate the other skew conditions were as follows: skew = 2, F(4,25); skew = 5, F(3,9); and skew = 10, F(10,7). The factorial crossing of sample size, effect size, and type of distribution resulted in 64 distinct parameter configurations. For each parameter configuration we generated 10,000 simulated samples. All simulations were performed by using SAS data step language (SAS, version 9.2). Results Within-Skew Findings Normal Distribution (Skew = 0) For the smallest sample size (N = 10), the mean difference, or empirical discrepancy, between the obtained sample correlation and the correlation estimated
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Hittner & May
71
by using equation (1) was quite small (
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study a
James B. Hittner & Kim May
a
a
College of Charleston Published online: 04 Apr 2012.
To cite this article: James B. Hittner & Kim May (2012) How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study, The Journal of General Psychology, 139:2, 68-77, DOI: 10.1080/00221309.2012.661376 To link to this article: http://dx.doi.org/10.1080/00221309.2012.661376
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [Umeå University Library] at 18:21 19 November 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
The Journal of General Psychology, 2012, 139(2), 68–77 C 2012 Taylor & Francis Group, LLC Copyright
Downloaded by [Umeå University Library] at 18:21 19 November 2014
How Accurate Is the Pearson r-from-Z Approximation? A Monte Carlo Simulation Study JAMES B. HITTNER KIM MAY College of Charleston
ABSTRACT. The Pearson r-from-Z approximation estimates the sample correlation (as an effect size measure) from the ratio of two quantities: the standard normal deviate equivalent (Z-score) corresponding to a one-tailed p-value divided by the square root of the total (pooled) sample size. The formula has utility in meta-analytic work when reports of research contain minimal statistical information. Although simple to implement, the accuracy of the Pearson r-from-Z approximation has not been empirically evaluated. To address this omission, we performed a series of Monte Carlo simulations. Results indicated that in some cases the formula did accurately estimate the sample correlation. However, when sample size was very small (N = 10) and effect sizes were small to small-moderate (ds of 0.1 and 0.3), the Pearson r-from-Z approximation was very inaccurate. Detailed figures that provide guidance as to when the Pearson r-from-Z formula will likely yield valid inferences are presented. Keywords: Pearson r-from-Z, correlation, meta-analysis, effect size, Monte Carlo simulation
WHEN PERFORMING A QUANTITATIVE RESEARCH SYNTHESIS, or metaanalysis, researchers rarely have access to raw data. Instead, the norm is to incorporate published summary statistics (e.g., t, F, χ 2, and odds ratio) into conversion formulas to compute standardized effect size estimates. Moreover, many of the common effect size measures, such as Cohen’s d (Cohen, 1988), Hedges’ g (Hedges, 1982), and the product-moment correlation, r, can be converted from one effect size metric to the other (see Cooper & Hedges, 1994, and Rosenthal & Rosnow, 1991, for relevant formulas). Although different researchers may prefer working with different effect size measures, Rosenthal (1991) argued that the product-moment correlation, r, has several desirable properties relative to other indices. First, unlike effect size measures based on mean differences in which Address correspondence to James B. Hittner, PhD, Department of Psychology, College of Charleston, 66 George Street, Charleston, SC 29424, USA; [email protected] (e-mail). 68
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Hittner & May
69
the sample size component varies based on whether observations are correlated or independent, the correlation coefficient, r, does not require any special adjustments. Second, relative to other effect size measures, the correlation arguably is more intuitively meaningful and can readily be interpreted in terms of practical utility. For example, the square of the correlation represents the amount of variance shared between two variables. In addition, researchers from various fields might wish to convert rs to binomial effect size displays (BESDs; Rosenthal, 1994) to demonstrate the real-world importance of an effect size indexed by a correlation coefficient. Although not without critics (Hsu, 2004), BESDs can be useful for gauging the improvement in success rate associated, for example, with initiating a new treatment procedure or incorporating a new predictor variable (Rosenthal, 1991; 1994). A third desirable property of r is its greater flexibility relative to other commonly used effect size measures, such as Cohen’s d. For example, r can always be computed from summary statistics whenever d can, but d cannot always be calculated whenever r can (see Rosenthal, 1991, p. 18). Although many published empirical research articles include the data necessary to compute correlation-based measures of effect size, some reports do not. In addition, research that is presented or summarized in other formats, such as conference presentations, poster handouts, published abstracts, press releases, and media reports often contain even less statistical information. These mediums for communicating research might emphasize sound bites and bullet points at the expense of statistical data. When data are presented, they often consist of p-values and group sample sizes only. In some reports, numerical data are absent altogether and the results merely are presented as either “statistically significant” or “non-significant.” For those research reports that do include specific p-values and group sample sizes, Rosenthal (1991) wrote in his popular meta-analysis book that researchers can estimate an effect size, r, by converting the obtained one-tailed p-value to its standard normal deviate equivalent (Z-score) and then computing r from the following formula, where N equals the total (pooled) sample size: Z r=√ N
(1)
Although Rosenthal’s simple conversion formula is convenient and easy to use, the formula’s accuracy has not, to our knowledge, been the subject of rigorous empirical evaluation. Delineating the conditions under which Rosenthal’s Pearson r-from-Z approximation accurately estimates the sample correlation will provide researchers with guidance as to when equation (1) will yield valid inferences. Given the absence of such empirically-driven guidance, the purpose of the present study was to examine, through a series of Monte Carlo simulations, the accuracy of equation (1) in estimating the sample correlation under a variety of sample size, distributional, and population effect size conditions.
70
The Journal of General Psychology
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Method The approach taken here required the creation of repeated sets of two independent samples with a specified skew and effect size. We note here that because the skew of the F-distribution is a direct function of the numerator and denominator degrees of freedom, it is relatively straightforward to specify an F-distribution with a particular value for skewness. This was accomplished by using SAS data step language in the following manner. First, two random variables, x1 and x2 , were generated from a uniform distribution (so that all values were equally likely). Then, a third variable, y1 , was created by transforming x1 to a quantile from an Fdistribution (i.e., a value of F) with the desired level of skewness. A fourth variable, y2 , was created by also transforming x2 to a quantile from an F-distribution with the specified skewness and then adding to y2 a multiple of the standard deviation of the given F-distribution, thereby adding the effect size in the form of a specified value of Cohen’s d. This procedure produced the two independent samples mentioned above. Aside from the effect size, each pair of y1 , y2 variables had the same level of skewness. In other words, we did not examine cases in which the two distributions have unequal skew. Then for each simulated data set (values of the simulated dependent variables, y1 and y2 ), a two independent-samples t-test on the means (of y1 vs. y2 ) was performed to generate a one-tailed p-value. From each one-tailed p, the corresponding standard normal deviate, or Z-score, was obtained and the estimated correlation from equation (1) was calculated. Then the actual observed correlation between the dependent variable (a single Y vector formed by concatenating y1 and y2 ) and the independent variable (a binary variable indexing group membership in y1 vs. y2 ) was computed. This is, of course, the observed effect size measure rIV,DV , which is estimated by equation (1). Finally, for each simulated sample, the numerical difference between the estimated correlation and the actual observed sample correlation was calculated and retained for analysis. Regarding the parameter specifications, we examined four different sample sizes (Ns = 10, 30, 60, and 100), four effect size conditions (ds = 1.0, 0.6, 0.3, and 0.1), and four different data distributions (skews = 0, 2, 5, and 10). The skew = 0 condition represents the normal distribution. The F distributions used to generate the other skew conditions were as follows: skew = 2, F(4,25); skew = 5, F(3,9); and skew = 10, F(10,7). The factorial crossing of sample size, effect size, and type of distribution resulted in 64 distinct parameter configurations. For each parameter configuration we generated 10,000 simulated samples. All simulations were performed by using SAS data step language (SAS, version 9.2). Results Within-Skew Findings Normal Distribution (Skew = 0) For the smallest sample size (N = 10), the mean difference, or empirical discrepancy, between the obtained sample correlation and the correlation estimated
Downloaded by [Umeå University Library] at 18:21 19 November 2014
Hittner & May
71
by using equation (1) was quite small (
