Confidence intervals and sample size calculations for the standardized mean difference effect size between two normal populations under heteroscedasticity

Abstract

The use of effect sizes and associated confidence intervals in all empirical research has been strongly emphasized by journal publication guidelines. To help advance theory and practice in the social sciences, this article describes an improved procedure for constructing confidence intervals of the standardized mean difference effect size between two independent normal populations with unknown and possibly unequal variances. The presented approach has advantages over the existing formula in both theoretical justification and computational simplicity. In addition, simulation results show that the suggested one-and two-sided confidence intervals are more accurate in achieving the nominal coverage probability. The proposed estimation method provides a feasible alternative to the most commonly used measure of Cohen's d and the corresponding interval procedure when the assumption of homogeneous variances is not tenable. To further improve the potential applicability of the suggested methodology, the sample size procedures for precise interval estimation of the standardized mean difference are also delineated. The desired precision of a confidence interval is assessed with respect to the control of expected width and to the assurance probability of interval width within a designated value. Supplementary computer programs are developed to aid in the usefulness and implementation of the introduced techniques. Kirk (1996), Kline (2004), Olejnik and Algina (2000), Richardson (1996), Rosenthal, Rosnow, and Rubin (2000), Rosenthal (2003), andVacha-Haase andThompson (2004). It has steadily become a general consensus in the methodological literature of behavior, education, management, and related disciplines that effect sizes accompanied by their corresponding confidence intervals are perhaps the best approach for conveying quantitative information in applied research.According to the general review of Ferguson (2009), effect sizes can be categorized into four general classes: (1) group difference, (2) strength of association, (3) corrected estimates, and (4) risk estimates. The group difference indices estimate the magnitude of difference between two or more groups, and Cohen's d (Cohen, 1969) is the most commonly used measure across virtually all disciplines of the social sciences. Specifically, Cohen's d is an estimate of the standardized mean difference that reflects the difference between two sample means divided by their pooled sample standard deviation under homoscedasticity. For the purpose of measuring the size of effect between two treatment groups with unequal variances, Cohen's d is no longer a proper estimator, because its standardizer, the Electronic supplementary material The online version of this article

Cohen’s d needs to be readily interpretable: Comment on Shieh (2013)

“…Noting the heteroscedasticity, our best estimate of that SD will be our sample SD for the control group, rather than an estimate pooled across experimental and control groups. Given heteroscedasticity, inference on the difference poses the challenges that Hedges (1981) and Shieh (2013) addressed, but choosing a standardizer for d should be no more difficult than preference for the control group SD over a pooled value (Grissom & Kim, 2012, p. 68). I suggest that researchers should prefer such a d to the d* discussed by Shieh.…”

“…3 reduces to 2σ, not σ, and so δ* = δ / 2. Therefore, in the equal-variance, equalgroup-sizes case, δ*, as used by Shieh (2013) and Kulinskaya and Staudte (2007), does not reduce to Cohen's δ as most commonly defined.…”

“…Only with a consistent and readily interpretable standardizer can d have the generality-across researchers, experimental designs, and situations-that is necessary for meaningful interpretation and inclusion in meta-analyses. Shieh (2013) recognized that "the choice of a suitable effect size parameter and statistic is a difficult and substantive decision" (p. 2). An important thread through the current comment is that, indeed, a range of substantive decisions are needed when selecting, calculating, and interpreting standardized ESs; informed judgment in context is required.…”

Why Hedges’ g*s based on the non-pooled standard deviation should be reported with Welch’s t-test

“…In "between-subject" designs where individuals are randomly assigned into one of two independent groups and group scores are compared based on their means, the dominant estimator of effect size is Cohen's d s , where the sample mean difference is divided by the pooled sample standard deviation (Peng, Chen, Chiang, & Chiang, 2013;Shieh, 2013).…”

“…However, the question which effect size to report when equal variances are not assumed has received less attention. One possible reason is that researchers have not found consensus on which of the available options should be used (Shieh, 2013). Even within the very specific context of an estimate for the standardized sample mean difference there is little agreement about which estimator is the best choice.…”