On the probability of making Type I errors.

Abstract

A statistical test leads to a Type I error whenever it leads to the rejection of a null hypothesis that is in fact true. The probability of making a Type I error can be characterized in the following three ways: the conditional prior probability (the probability of making a Type I error whenever a true null hypothesis is tested), the overall prior probability (the probability of making a Type I error across all experiments), and the conditional posterior probability (the probability of having made a Type I error in situations in which the null hypothesis is rejected). In this article, we show (a) that the alpha level can be equated with the first of these and (b) that it provides an upper bound for the second but (c) that it does not provide an estimate of the third, although it is commonly assumed to do so. We trace the source of this erroneous assumption first to statistical texts used by psychologists, which are generally ambiguous about which of the three interpretations is intended at any point in their discussions of Type I errors and which typically confound the conditional price and posterior probabilities. Underlying this, however, is a more general fallacy in reasoning about probabilities, and we suggest that this may be the result of erroneous inferences about probabilistic conditional statements. Finally, we consider the possibility of estimating the (posterior) probability of a Type I error in situations in which the null hypothesis is rejected and, hence, the proportion of statistically significant results that may be Type I errors.We are grateful to Larry Phillips for his comments on a previous version of this article and to an anonymous reviewer for bringing the article by Carver (1978) to our attention.

When null hypothesis significance testing is unsuitable for research: a reassessment

“…Most textbooks illustrate NHST by partial 2 × 2 tables (see Table 1) which fail to contextualize long-run conditional probabilities and fail to clearly distinguish between long-run probabilities and the p-value which is computed for a single data set (Pollard and Richardson, 1987). This leads to major confusions about the meaning of the p-value (see Appendix 2 in Supplementary Material).…”

“…That is, researchers are interested in the post-experimental probability of H 0 and H 1 . Most probably, for the reason that researchers do not get what they really want to see (Murdoch et al, 2008 and the only parameter NHST computes is the p-value it is welldocumented (Oakes, 1986;Gliner et al, 2002;Castro Sotos et al, 2007Wilkerson and Olson, 2010;Hoekstra et al, 2014) that many, if not most researchers confuse FRP with the p-value or α and they also confuse the complement of p-value (1-p) or α (1-α) with TRP (Pollard and Richardson, 1987;Cohen, 1994). These confusions are of major portend because the difference between these completely different parameters is not minor, they can differ by orders of magnitude, the long-run FRP being much larger than the p-value under realistic conditions (Sellke et al, 2001;Ioannidis, 2005).…”

“…Simply put, the p-value is pretty much the only thing that NHST computes but scientists usually would like to know the probability of their theory being true or false in light of their data (Pollard and Richardson, 1987;Goodman, 1993;Jaynes, 2003;Wagenmakers, 2007). That is, researchers are interested in the post-experimental probability of H 0 and H 1 .…”

“…However, statistical significance does not protect us from false positives. FRP can only be minimized by keeping H 0 :H 1 odds and bias low and power high (Pollard and Richardson, 1987;Button et al, 2013;Bayarri et al, 2016). Hence, power is not only important so that we increase our chances to detect true effects but it is also crucial in keeping FRP low.…”

“…The modeling supporting this claim refers to the long-run FRP and TRP which we can compute by applying Bayes' theorem (see Figure 4 for illustration, see computational details and further illustration in Appendix 3 in Supplementary Material). The calculations must consider α, the power (1-β) of the statistical test used, the prestudy probabilities of H 0 and H 1 , and it is also insightful to consider bias (Berger, 1985;Berger and Delampady, 1987;Berger and Sellke, 1987;Pollard and Richardson, 1987;Lindley, 1993;Sellke et al, 2001;Sterne and Smith, 2001;Ioannidis, 2005). While NHST neglects the pre-study odds of H 0 and H 1 , these are crucial to take into account when calculating FRP and TRP.…”