Effect Size Reporting Practices in Published Articles

Alhija, Fadia Nasser-Abu; Levy, Aviad

doi:10.1177/0013164408315266

Cited by 37 publications

(61 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…See Table 4 for details. This result is consistent with the findings in the previous studies (e.g., Alhija & Levy, 2007;Hutchins & Henson, 2002;Kirk, 1996;McMillan et al, 2002), though Dunleavy et al (2006) found that variance-account-for statistics were typically omitted. The popularity of this type of effect size measure can be explained by the fact that 76% of the 1,243 articles (n ϭ 938) used general linear models as the main NHST method, and 75% of the 610 articles that reported effect size (n ϭ 455) used general linear models (cf.…”

Section: Reporting Effect Sizesupporting

confidence: 92%

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Sun¹,

Pan²,

Wang³

2010

Journal of Educational Psychology

245

146

View full text Add to dashboard Cite

Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic associations. As a measure of practical significance, effect size quantifies the size of mean differences or strength of associations and directly answers the research questions. Furthermore, a comparison of effect sizes across studies facilitates meta-analytic assessment of the effect size and accumulation of knowledge. In the current comprehensive review, we investigated the most recent effect size reporting and interpreting practices in 1,243 articles published in 14 academic journals from 2005 to 2007. Overall, 49% of the articles reported effect size-57% of which interpreted effect size. As an empirical study for the sake of good research methodology in education and psychology, in the present study we provide an illustrative example of reporting and interpreting effect size in a published study. Furthermore, a 7-step guideline for quantitative researchers is also summarized along with some recommended resources on how to understand and interpret effect size. Burnham, & Thompson, 2000;Kirk, 1996;Plucker, 1997;Robinson & Levin, 1997;Thompson, 1998bThompson, , 1999cThompson, , 1999dThompson & Snyder, 1997). The effectiveness of this recommendation is worthy of a methodological review. Thus, the purpose of the present study was to investigate the effect size reporting and interpreting practices in academic journals in education and psychology areas. To build the present study into a solid theoretical framework, we review the problems and misuses of NHST, and we emphasize the importance of effect size and its relationship with NHST, confidence intervals, and meta-analytical assessment of effect size. An illustrative example of reporting and interpreting effect size in a published study is also presented. To facilitate quantitative researchers' use of effect size, we recommend resources on how to understand and use effect size, and we provide a seven-step guideline for quantitative researchers to conclude the study. Keywords NHSTNHST is a traditional and popular approach to make statistical inference about research questions (Anderson et al., 2000). It is considered to be an objective, scientific procedure for knowledge accumulation (Kirk, 1996). It frames research questions in terms of two contrasting statistical hypotheses. For instance, when the purpose is to examine the effect of a treatment, the null hypothesis states that "the experimental group and the control group are not different with respect to [a specified property of interest] and that any difference found between their means is due to sampling fluctuation" (Carver, 1978, p. 381), whereas the alternative hypothesis states the opposite for a two-tailed test of mean difference as population parameter. When...

show abstract

Section: Reporting Effect Sizesupporting

confidence: 92%

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Sun¹,

Pan²,

Wang³

2010

Journal of Educational Psychology

245

146

View full text Add to dashboard Cite

show abstract

“…According to the editorial guidelines and methodological recommendations of several prominent educational and psychological journals, it is necessary to include some measures of effect size and confidence intervals for all primary outcomes (Alhija & Levy, 2009;Fritz, Morris & Richler, 2012;Kelley & Preacher, 2012;Odgaard & Fowler, 2010;Peng et al, 2013;Sun, Pan & Wang, 2010). Among the several inadequate effect size reporting and interpretation practices, Alhija and Levy (2009) and Peng et al (2013) especially emphasized that the majority of popular effect size measures are positively biased estimators such as the standardized mean difference index Cohen's d, the strength of association measureη 2 , and the sample squared multiple correlation coefficient R 2 .…”

Section: Discussionmentioning

confidence: 99%

Choosing the best index for the average score intraclass correlation coefficient

Shieh

2015

Behav Res

View full text Add to dashboard Cite

The intraclass correlation coefficient (ICC)(2) index from a one-way random effects model is widely used to describe the reliability of mean ratings in behavioral, educational, and psychological research. Despite its apparent utility, the essential property of ICC(2) as a point estimator of the average score intraclass correlation coefficient is seldom mentioned. This article considers several potential measures and compares their performance with ICC(2). Analytical derivations and numerical examinations are presented to assess the bias and mean square error of the alternative estimators. The results suggest that more advantageous indices can be recommended over ICC(2) for their theoretical implication and computational ease.

show abstract

“…Correspondingly, numerous practical guidelines and suggestions for selecting, calculating, and interpreting effect size indices for various types of statistical analyses have been provided in the literature, such as Alhija and Levy (2009), Breaugh (2003), Durlak (2009), Ferguson (2009, Fern and Monroe (1996), Grissom and Kim (2012), Huberty (2002), Kirk (1996), Kline (2004), Olejnik and Algina (2000), Richardson (1996), Rosenthal, Rosnow, and Rubin (2000), Rosnow and Rosenthal (2003), and Vacha-Haase and Thompson (2004). It has steadily become a general consensus in the methodological literature of behavior, education, management, and related disciplines that effect sizes accompanied by their corresponding confidence intervals are perhaps the best approach for conveying quantitative information in applied research.…”

mentioning

confidence: 99%

Confidence intervals and sample size calculations for the standardized mean difference effect size between two normal populations under heteroscedasticity

Shieh

2013

Behav Res

View full text Add to dashboard Cite

The use of effect sizes and associated confidence intervals in all empirical research has been strongly emphasized by journal publication guidelines. To help advance theory and practice in the social sciences, this article describes an improved procedure for constructing confidence intervals of the standardized mean difference effect size between two independent normal populations with unknown and possibly unequal variances. The presented approach has advantages over the existing formula in both theoretical justification and computational simplicity. In addition, simulation results show that the suggested one-and two-sided confidence intervals are more accurate in achieving the nominal coverage probability. The proposed estimation method provides a feasible alternative to the most commonly used measure of Cohen's d and the corresponding interval procedure when the assumption of homogeneous variances is not tenable. To further improve the potential applicability of the suggested methodology, the sample size procedures for precise interval estimation of the standardized mean difference are also delineated. The desired precision of a confidence interval is assessed with respect to the control of expected width and to the assurance probability of interval width within a designated value. Supplementary computer programs are developed to aid in the usefulness and implementation of the introduced techniques. Kirk (1996), Kline (2004), Olejnik and Algina (2000), Richardson (1996), Rosenthal, Rosnow, and Rubin (2000), Rosenthal (2003), andVacha-Haase andThompson (2004). It has steadily become a general consensus in the methodological literature of behavior, education, management, and related disciplines that effect sizes accompanied by their corresponding confidence intervals are perhaps the best approach for conveying quantitative information in applied research.According to the general review of Ferguson (2009), effect sizes can be categorized into four general classes: (1) group difference, (2) strength of association, (3) corrected estimates, and (4) risk estimates. The group difference indices estimate the magnitude of difference between two or more groups, and Cohen's d (Cohen, 1969) is the most commonly used measure across virtually all disciplines of the social sciences. Specifically, Cohen's d is an estimate of the standardized mean difference that reflects the difference between two sample means divided by their pooled sample standard deviation under homoscedasticity. For the purpose of measuring the size of effect between two treatment groups with unequal variances, Cohen's d is no longer a proper estimator, because its standardizer, the Electronic supplementary material The online version of this article

show abstract

Effect Size Reporting Practices in Published Articles

Cited by 37 publications

References 33 publications

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Choosing the best index for the average score intraclass correlation coefficient

Confidence intervals and sample size calculations for the standardized mean difference effect size between two normal populations under heteroscedasticity

Contact Info

Product

Resources

About