The Interpretation of Levels of Significance by Psychological Researchers

Rosenthal, Robert; Gaito, John

doi:10.1080/00223980.1963.9916596

Cited by 151 publications

(114 citation statements)

References 3 publications

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…Peer reviewers should help authors avoid such mistakes. The statistical error may also be a manifestation of the cliff effect 7 , the phenomenon that many people's confidence in a result drops abruptly when a P value increases just beyond the 0.05 level. Indeed, people are generally tempted to attribute too much meaning to the difference between significant and not significant.…”

Section: P E R S P E C T I V Ementioning

confidence: 99%

Erroneous analyses of interactions in neuroscience: a problem of significance

2011

View full text Add to dashboard Cite

In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling."The percentage of neurons showing cue-related activity increased with training in the mutant mice (P < 0.05), but not in the control mice (P > 0.05)." "Animals receiving vehicle (control) infusions into the amygdala showed increased freezing to the conditioned stimulus compared with a control stimulus (P < 0.01); in animals receiving muscimol infusions into the amygdala, this difference was abolished (F < 1)."These two fictive, but representative, statements illustrate a statistical error that is common in the neuroscience literature. The researchers who made these statements wanted to claim that one effect (for example, the training effect on neuronal activity in mutant mice) was larger or smaller than the other effect (the training effect in control mice). To support this claim, they needed to report a statistically significant interaction (between amount of training and type of mice), but instead they reported that one effect was statistically significant, whereas the other effect was not. Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significant 1 . Consider an extreme scenario in which traininginduced activity barely reaches significance in mutant mice (for example, P = 0.049) and barely fails to reach significance for control mice (for example, P = 0.051). Despite the fact that these two P values lie on opposite sides of 0.05, one cannot conclude that the training effect for mutant mice differs statistically from that for control mice.That is, as famously noted by Rosnow and Rosenthal 2 , "surely, God loves the 0.06 nearly as much as the 0.05". Thus, when making a comparison between two effects, researchers should report the statistical significance of their difference rather than the difference between their significance levels.Our impression was that this error of comparing significance levels is widespread in the neuroscience literature, but until now there were no aggregate data to support this impression. We therefore examined all of the behavioral, systems and cognitive neuroscience studies published in four prestigious journals (Nature, Science, Nature Neuroscience and Neuron) ...

show abstract

Section: P E R S P E C T I V Ementioning

confidence: 99%

Erroneous analyses of interactions in neuroscience: a problem of significance

2011

View full text Add to dashboard Cite

show abstract

“…Although the focus of the significance testing debate has centered on the theoretical arguments for or against it, some research has focused on the interpretation of significance testing outcomes in practice (e.g., Lecoutre et al, 2003;Oakes, 1986;Rosenthal & Gaito, 1963;Weisburd, Lum, & Yang, 2003). For example, Finch et al (2001) focused on misconceptions about significance testing as revealed in published reports.…”

mentioning

confidence: 99%

Probability as certainty: Dichotomous thinking and the misuse ofp values

et al. 2006

View full text Add to dashboard Cite

“…Atlhough sample size is certainly a factor that should be considered in evaluating our results, Keppel (1991) has discussed the issue of small versus large sample size and has indicated that a small sample size is not necessarily problematical. Many researchers erroneously believe that the results of an experiment having a small sample size and an Fvalue significant at p < .05 are less important or significant than the results of an experiment having a larger sample size and an F significant at p < .05 (Rosenthal & Gaito, 1963). However, Keppel (1991) pOints out that the study with the smaller sample size is likely to have the more substantial effect.…”

Section: Discussionmentioning

confidence: 94%