The Case Against Statistical Significance Testing

Carver, Ronald P.

doi:10.17763/haer.48.3.t490261645281841

Cited by 804 publications

(595 citation statements)

References 22 publications

Supporting

Mentioning

569

Contrasting

Unclassified

Order By: Relevance

“…The earliest serious challenges to NHST dated back to 1938 when Joseph Berkson published his article to question the logic and usefulness of NHST (Berkson, 1938). Since then, criticisms of NHST have noticeably intensified (e.g., Anderson et al, 2000;Carver, 1978;Cohen, 1990Cohen, , 1994Falk & Greenbaum, 1995;Harlow, Mulaik, & Steiger, 1997;Henson & Smith, 2000;Katzer & Sodt, 1973;Kirk, 1996;Robinson & Wainer, 2002;Schmidt, 1996;Yates, 1951).…”

Section: Problems With Nhstmentioning

confidence: 99%

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Sun¹,

Pan²,

Wang³

2010

Journal of Educational Psychology

247

146

View full text Add to dashboard Cite

Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic associations. As a measure of practical significance, effect size quantifies the size of mean differences or strength of associations and directly answers the research questions. Furthermore, a comparison of effect sizes across studies facilitates meta-analytic assessment of the effect size and accumulation of knowledge. In the current comprehensive review, we investigated the most recent effect size reporting and interpreting practices in 1,243 articles published in 14 academic journals from 2005 to 2007. Overall, 49% of the articles reported effect size-57% of which interpreted effect size. As an empirical study for the sake of good research methodology in education and psychology, in the present study we provide an illustrative example of reporting and interpreting effect size in a published study. Furthermore, a 7-step guideline for quantitative researchers is also summarized along with some recommended resources on how to understand and interpret effect size. Burnham, & Thompson, 2000;Kirk, 1996;Plucker, 1997;Robinson & Levin, 1997;Thompson, 1998bThompson, , 1999cThompson, , 1999dThompson & Snyder, 1997). The effectiveness of this recommendation is worthy of a methodological review. Thus, the purpose of the present study was to investigate the effect size reporting and interpreting practices in academic journals in education and psychology areas. To build the present study into a solid theoretical framework, we review the problems and misuses of NHST, and we emphasize the importance of effect size and its relationship with NHST, confidence intervals, and meta-analytical assessment of effect size. An illustrative example of reporting and interpreting effect size in a published study is also presented. To facilitate quantitative researchers' use of effect size, we recommend resources on how to understand and use effect size, and we provide a seven-step guideline for quantitative researchers to conclude the study. Keywords NHSTNHST is a traditional and popular approach to make statistical inference about research questions (Anderson et al., 2000). It is considered to be an objective, scientific procedure for knowledge accumulation (Kirk, 1996). It frames research questions in terms of two contrasting statistical hypotheses. For instance, when the purpose is to examine the effect of a treatment, the null hypothesis states that "the experimental group and the control group are not different with respect to [a specified property of interest] and that any difference found between their means is due to sampling fluctuation" (Carver, 1978, p. 381), whereas the alternative hypothesis states the opposite for a two-tailed test of mean difference as population parameter. When...

show abstract

Section: Problems With Nhstmentioning

confidence: 99%

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Sun¹,

Pan²,

Wang³

2010

Journal of Educational Psychology

247

146

View full text Add to dashboard Cite

show abstract

“…Small differences can be statistically ''significant'' simply because of a large sample size. This has prompted some to argue that tests of statistical significance are not generally useful (Carver, 1978(Carver, , 1993Cohen, 1994;Hunter, 1997;Kirk, 1996;Schmidt, 1992). Rather, these critics of significance testing argue that confidence intervals and measures of effect size should be the focal point of research findings.…”

mentioning

confidence: 99%

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

Olejnik¹,

Algina

2000

Contemporary Educational Psychology

576

370

View full text Add to dashboard Cite

Although dissatisfaction with the limitations associated with tests for statistical significance has been growing for several decades, applied researchers have continued to rely almost exclusively on these indicators of effect when reporting their findings. To encourage an increased use of alternative measures of effect, the present paper discusses several measures of effect size that might be used in group comparison studies involving univariate and/or multivariate models. For the methods discussed, formulas are presented and data from an experimental study are used to demonstrate the application and interpretation of these indices. The paper concludes with some cautionary notes on the limitations associated with these measures of effect size.

show abstract

“…Such approaches to analysis take no account of sample quality or attrition (Lipsey et al 2012), being predicated on complete random samples of a kind never encountered in real-life research (Berk and Freedman 2001). They do not make sense anyway (Carver 1978), are routinely misinterpreted (Watts 1991), and can lead to serious mistakes for policy and practice (Falk and Greenbaum 1995).…”

Section: A Summary Of Evidence On the Effectiveness Of The General Phmentioning

confidence: 99%

An evaluation of Fresh Start as a catch-up intervention: a trial conducted by teachers

Gorard

Siddiqui

See

2016

Educational Studies

View full text Add to dashboard Cite

Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. AbstractThis paper describes a randomised controlled trial conducted with 10 secondary schools in England to evaluate the impact and feasibility of Fresh Start as an intervention to help new entrants with low prior literacy. Fresh Start is a synthetic phonics programme for small groups of pupils, here implemented three times per week over 22 weeks. The intervention was led by the schools themselves and assessed in co-operation with the independent evaluators. A group of 433 Year 7 pupils (first year of secondary school) were identified by schools as having literacy attainment below 'secure' KS2 Level 4, and individually randomised to a treatment group or a waiting-list control. The pupils were assessed via GL's New Group Reading Test. Missing data at pre-and post-test amounted to 3% of the total. The overall 'effect' size in terms of gain scores from pre-to post-test was +0.24, and this was repeated in a sub-group analysis involving only FSM-eligible pupils. However, there was some imbalance between the two groups at the outset, and this must lead to a slight caution about the findings, and to some doubts about whether one or more schools unwittingly subverted the randomisation. Other than this, the aggregated trial shows that schools can conduct evaluations of their own interventions with firm guidance from experts, and under favourable conditions such as individual randomisation and lack of extended involvement by developers with a conflict of interests.

show abstract

The Case Against Statistical Significance Testing

Cited by 804 publications

References 22 publications

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

An evaluation of Fresh Start as a catch-up intervention: a trial conducted by teachers

Contact Info

Product

Resources

About