The inaccuracy of sample-based confidence intervals to estimate a priori ones

Abstract

Confidence intervals (CIs) constitute the most popular alternative to widely criticized null hypothesis significance tests. CIs provide more information than significance tests and lend themselves well to visual displays. Although CIs are no better than significance tests when used solely as significance tests, researchers need not limit themselves to this use of CIs. Rather, CIs can be used to estimate the precision of the data, and it is the precision argument that may set CIs in a superior position to significance tests. We tested two versions of the precision argument by performing computer simulations to test how well sample-based CIs estimate a priori CIs. One version pertains to precision of width whereas the other version pertains to precision of location. Using both versions, sample-based CIs poorly estimate a priori CIs at typical sample sizes and perform better as sample sizes increase.

Probabilistic thinking is the name of the game: Integrating test and confidence intervals to plan sample sizes

“…Unfortunately, misunderstandings of CIs are not uncommon, and further guidance regarding their use and interpretation is still needed (Belia et al, 2005;Cumming & Calin-Jageman, 2017;Fidler et al, 2004;Morey et al, 2016). Namely, asking how much and how uncertain are the salient statistical literacy (Calin-Jageman & Cumming, 2019;Trafimow, 2018;Trafimow & Uhalt, 2020). In particular, it should be noted that the sample size calculation for constructing CIs may be different from those for testing hypotheses (Borenstein et al, 2001) because the expected width or precision is the concern, rather than the statistical power (Kelley et al, 2003).…”

The APP procedure for estimating the Cohen's effect size

“…This last is unknown. In addition, Trafimow and Uhalt (2020) have shown that sample confidence intervals tend to be inaccurate representations of population confidence intervals unless sample sizes are much larger than those typically employed. An alternative way to address the issue is to use the a priori procedure (APP) that has been employed previously in a variety of ways not pertaining to Cohen's d (e.g.…”

Mandatory theorizing and synthesis: A comment on Phaf (2020)

“…In contrast, however, Trafimow (2018b) has shown that confidence intervals are problematic in that they confound three kinds of precision. Trafimow and Uhalt (in press) demonstrated that traditional confidence intervals based on samples are very poor estimates of population intervals unless sample sizes are ridiculously large. Therefore, traditional confidence intervals are not useful either.…”