The fallacy of placing confidence in confidence intervals

Abstract

Interval estimates – estimates of parameters that include an allowance for sampling uncertainty – have long been touted as a key component of statistical analyses. There are several kinds of interval estimates, but the most popular are confidence intervals (CIs): intervals that contain the true parameter value in some known proportion of repeated samples, on average. The width of confidence intervals is thought to index the precision of an estimate; CIs are thought to be a guide to which parameter values are plausible or reasonable; and the confidence coefficient of the interval (e.g., 95 %) is thought to index the plausibility that the true parameter is included in the interval. We show in a number of examples that CIs do not necessarily have any of these properties, and can lead to unjustified or arbitrary inferences. For this reason, we caution against relying upon confidence interval theory to justify interval estimates, and suggest that other theories of interval estimation should be used instead. Electronic supplementary material The online version of this article (doi:10.3758/s13423-015-0947-8) contains supplementary material, which is available to authorized users.

Keep Calm and Learn Multilevel Logistic Modeling: A Simplified Three-Step Procedure Using Stata, R, Mplus, and SPSS

“…In doing so, Stata users have to add, cov(uns) at the end of the command; R users have to replace (1 + lvl1_predict || id_cluster) by (1 + lvl1_predict | id_cluster) in the glmer function; Mplus users have to add s1 WITH outcome (s1 is the name given to the random slope) to the %BETWEEN% part of the model; and SPSS users have to specify COVARIANCE_TYPE = UNSTRUCTURED in the com- (Gelman & Hill, 2007). 9 Formally speaking, the interpretation of the confidence interval is as such: If we repeated the study an infinite number of time, and computed a 95% confidence interval each time, then 95% of these confidence intervals would contain the true population odds ratio and 5% of them would miss it (see Morey, Hoekstra, Rouder, Lee & Wagenmakers, 2016). 10 The standard errors of the simple slopes should normally be calculated using the Delta method (Greene, 2010); this method is not covered herein for the sake of simplicity.…”

ζ2Reticuli, its debris disk, and its lonely stellar companionζ1Ret

“…Since both approaches are addressing and solving exactly the same problem, the slopes calculated by the two methods are almost identical. However, since the CI (in frequentist analysis) and the HPD (in Bayesian approach) have different meanings, their values can be different and should be interpreted differently (e.g., Morey et al 2015). A 95% CI is an interval that in repeated sampling has a 0.95 probability of containing the true value of the parameter, i.e, if a large number of samples are used, the true value of the slope will fall within the CI in 95% of the cases.…”

Credible Confidence: A pragmatic view on the frequentist vs Bayesian debate

“…In general, the criticism of Bayesian methods is that there is too much room for subjectivity (or sometimes not enough, cf. Gelman, 2008), whereas the criticism to frequentist methods is that they are prone to misinterpretation (Bakan, 1966;Cohen, 1994;Goodman, 2008;Morey, Hoekstra, Rouder, Lee, & Wagenmakers, 2016;Oakes, 1986;Schervish, 1996) and provide answers to unasked questions (Wagenmakers, Lee, Lodewyckx, & Iverson, 2008).…”