Scientific method: Statistical errors

Nuzzo, Regina

doi:10.1038/506150a

Cited by 1,275 publications

(668 citation statements)

References 11 publications

Supporting

Mentioning

627

Contrasting

Unclassified

Order By: Relevance

“…Yet a fundamental problem inherent in standard statistical methods, one that is pervasively linked to the lack of reproducibility in research, remains to be considered: the wide sample-to-sample variability in the P value. This omission reflects a general lack of awareness about this crucial issue, and we address this matter here.Focusing on the P value during statistical analysis is an entrenched culture [11][12][13] . The P value is often used without the realization that in most cases the statistical power of a study is too low for P to assist the interpretation of the data (Box 1).…”

mentioning

confidence: 99%

See 1 more Smart Citation

The fickle P value generates irreproducible results

et al. 2015

View full text Add to dashboard Cite

The reliability and reproducibility of science are under scrutiny. However, a major cause of this lack of repeatability is not being considered: the wide sample-to-sample variability in the P value. We explain why P is fickle to discourage the ill-informed practice of interpreting analyses based predominantly on this statistic.Reproducible research findings are a cornerstone of the scientific method, providing essential validation. There has been recent recognition, however, that the results of published research can be difficult to replicate 1-7 , an awareness epitomized by a series in Nature entitled "Challenges in irreproducible research" and by the Reproducibility Initiative, a project intended to identify and reward reproducible research (http:// validation.scienceexchange.com/#/ reproducibilityinitiative). In a recent meeting at the American Association for the Advancement of Science headquarters involving many of the major journals reporting biomedical science research, a common set of principles and guidelines was agreed upon for promoting transparency and reproducibility 8 . These discussions and initiatives all focused on a number of issues, including aspects of statistical reporting 9 , levels of statistical power (i.e., sufficient statistical capacity to find an effect; a 'statistically significant' finding) 10 and inclusion-exclusion criteria. Yet a fundamental problem inherent in standard statistical methods, one that is pervasively linked to the lack of reproducibility in research, remains to be considered: the wide sample-to-sample variability in the P value. This omission reflects a general lack of awareness about this crucial issue, and we address this matter here.Focusing on the P value during statistical analysis is an entrenched culture [11][12][13] . The P value is often used without the realization that in most cases the statistical power of a study is too low for P to assist the interpretation of the data (Box 1). Among the many and varied reasons for a fearful and hidebound approach to statistical practice, a lack of understanding is prominent 14 . A better understanding of why P is so unhelpful should encourage scientists to reduce their reliance on this misleading concept.Readers may know of the long-standing philosophical debate about the value and validity of null-hypothesis testing [15][16][17] . Although the P value formalizes null-hypothesis testing, this article will not revisit these issues. Rather, we concentrate on how P values themselves are misunderstood.Although statistical power is a central element in reliability 18 , it is often considered only when a test fails to demonstrate a real effect (such as a difference between groups): a 'false negative' result (see Box 2 for a glossary of statistical terms used in this article). Many scientists who are not statisticians do not realize that the power of a test is equally relevant when considering statistically significant results, that is, when the null hypothesis appears to be untenable. This is because the statistical power o...

show abstract

mentioning

confidence: 99%

“…Focusing on the P value during statistical analysis is an entrenched culture [11][12][13] . The P value is often used without the realization that in most cases the statistical power of a study is too low for P to assist the interpretation of the data (Box 1).…”

mentioning

confidence: 99%

The fickle P value generates irreproducible results

et al. 2015

View full text Add to dashboard Cite

show abstract

“…1,2 A less appreciated consequence of lower power is that a statistically significant result is more likely to be a false positive than a true effect. [2][3][4] The probability that a significant result is a false positive increases because a p-value less than 0.05 means either (1) an effect is present, or (2) no effect is present but an extreme (unlikely) result occurred by chance. We never know which of these options is correct, but with low power, true effects are harder to detect, and so the "effect present" option becomes less probable and the "chance occurrence" option becomes more probable.…”

Section: Introductionmentioning

confidence: 99%

Four simple ways to increase power without increasing the sample size

Lazic

2017

Preprint

View full text Add to dashboard Cite

Underpowered experiments have three problems: the probability of a false positive result is higher, true effects are harder to detect, and the true effects that are detected tend to have inflated effect sizes. Many biology experiments are underpowered and recent calls to change the traditional 0.05 significance threshold to a more stringent value of 0.005 will further reduce the power of the average experiment. Increasing power by increasing the sample size is often the only option considered, but more samples increases costs, makes the experiment harder to conduct, and is contrary to the 3Rs principles for animal research. We show how the design of an experiment and some analytical decisions can have a surprisingly large effect on power.

show abstract

“…Since statistical significance has been questioned for behavioural studies [25], effect sizes were investigated, as suggested by Coe [23], regarding the differences between every two experience groups at a time, in each trial block, for all indices.…”

Section: Human Movementmentioning

confidence: 99%

Analysis of previous perceptual and motor experience in breaststroke kick learning

Ried¹,

Rodrigues²,

Meira³

2018

Hum Mov.

View full text Add to dashboard Cite

One of the variables that influence motor learning is the learner's previous experience, which may provide perceptual and motor elements to be transferred to a novel motor skill. For swimming skills, several motor experiences may prove effective. Purpose. The aim was to analyse the influence of previous experience in playing in water, swimming lessons, and music or dance lessons on learning the breaststroke kick. Methods. The study involved 39 Physical Education students possessing basic swimming skills, but not the breaststroke, who performed 400 acquisition trials followed by 50 retention and 50 transfer trials, during which stroke index as well as rhythmic and spatial configuration indices were mapped, and answered a yes/no questionnaire regarding previous experience. Data were analysed by ANOVA (p = 0.05) and the effect size (Cohen's d 0.8 indicating large effect size).Results. The whole sample improved their stroke index and spatial configuration index, but not their rhythmic configuration index. Although differences between groups were not significant, two types of experience showed large practical effects on learning: childhood water playing experience only showed major practically relevant positive effects, and no experience in any of the three fields hampered the learning process. Conclusions. The results point towards diverse impact of previous experience regarding rhythmic activities, swimming lessons, and especially with playing in water during childhood, on learning the breaststroke kick.

show abstract

Scientific method: Statistical errors

Cited by 1,275 publications

References 11 publications

The fickle P value generates irreproducible results

The fickle P value generates irreproducible results

Four simple ways to increase power without increasing the sample size

Analysis of previous perceptual and motor experience in breaststroke kick learning

Contact Info

Product

Resources

About