Confirmatory Methods, or Huge Samples, Are Required to Obtain Power for the Evaluation of Theories

Abstract

Experimental studies are usually designed with specific expectations about the results in mind. However, most researchers apply some form of omnibus test to test for any differences, with follow up tests like pairwise comparisons or simple effects analyses for further investigation of the effects. The power to find full support for the theory with such an exploratory approach which is usually based on multiple testing is, however, rather disappointing. With the simulations in this paper we showed that many of the common choices in hypothesis testing led to a severely underpowered form of theory evaluation. Furthermore, some less commonly used approaches were presented and a comparison of results in terms of power to find support for the theory was made. We concluded that confirmatory methods are required in the context of theory evaluation and that the scientific literature would benefit from a clearer distinction between confirmatory and exploratory findings. Also, we emphasis the importance of reporting all tests, significant or not, including the appropriate sample statistics like means and standard deviations. Another recommendation is related to the fact that researchers, when they discuss the conclusions of their own study, seem to underestimate the role of sampling variability. The execution of more replication studies in combination with proper reporting of all results provides insight in between study variability and the amount of chance findings.

Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

“…Such a full report allows the reader to assess the robustness of the statistical evidence. Of course, the royal road to obtaining sufficient power is not to choose a lenient correction method; instead, one is best advised to plan for a large sample size (Klugkist, Post, Haarhuis, & van Wesel, 2014). …”

The power of informative hypotheses

“…The three approaches presented in this paper make use of sampling distributions of the Bayes factors under H i and H c , or under H i and H i . Approach 1, like in Klugkist et al (2014) and Weiss (1997), chooses…”

“…Sample size determination for the evaluation of the null hypothesis H 0 with an inequality constrained hypothesis H i using BF i0 is considered by Klugkist et al (2014). The decision criterion used is that Bayes factors larger and smaller than 1 result in conclusions in favor of H i and H 0 respectively.…”

“…The decision criterion used is that Bayes factors larger and smaller than 1 result in conclusions in favor of H i and H 0 respectively. Using this decision criterion, the sample size is determined for various effect sizes, such that the traditional Type I error probability is below .05, and the power is above .80 (Klugkist et al, 2014). Although this article uses order constrained hypotheses, no elaboration is made on the sample sizes required for the evaluation of H i with H c or with H i .…”

“…Unconditional error probabilities have been investigated in the context of Bayesian hypothesis testing (e.g. Weiss, 1997;Klugkist, Post, Haarhuis, & van Wesel, 2014).…”