Statistical Evidence in Experimental Psychology

Abstract

Statistical inference in psychology has traditionally relied heavily on p-value significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes. The second is to replace inference with Bayesian measures of evidence, such as the Bayes factor. The authors provide a practical comparison of p values, effect sizes, and default Bayes factors as measures of statistical evidence, using 855 recently published t tests in psychology. The comparison yields two main results. First, although p values and default Bayes factors almost always agree about what hypothesis is better supported by the data, the measures often disagree about the strength of this support; for 70% of the data sets for which the p value falls between .01 and .05, the default Bayes factor indicates that the evidence is only anecdotal. Second, effect sizes can provide additional evidence to p values and default Bayes factors. The authors conclude that the Bayesian approach is comparatively prudent, preventing researchers from overestimating the evidence in favor of an effect.

Bayesian hypothesis testing for human threat conditioning research: an introduction and the condir R package

“…If the user desires an interpretation of the results, they can click the ‘TRUE’ option in the ‘Should the results be interpreted?’ question. This will extend the results report (not shown here) so that the p -values are now interpreted as significant or non-significant, and the Bayesian results are interpreted based on the categories that are suggested in the literature (for more information on Bayes factor categories, see Jeffreys, 1961, and Wetzels et al, 2011). We note that this is a report based on the numerical values.…”

Control of information in working memory: Encoding and removal of distractors in the complex-span paradigm

“…This reflects the fact that Bayes Factors increase steeply with increasing set size once the set size exceeds ambiguous values (see Wetzels et al, 2011, for illustrative examples) …”

Visual working memory declines when more features must be remembered for each object

“…First, we analyzed our data with conventional ANOVAs and obtained highly significant main effects and clearly nonsignificant interactions in all three experiments, exactly matching the conclusions from the BANOVAs. Second, it has been shown that Bayesian t-tests using the JZS default prior are more conservative than conventional t-tests or ANOVAs (Rouder et al, 2009;Wetzels et al, 2011).…”