An exposition of the false confidence theorem

Abstract

A recent paper (Balch et al. (2017), 'Satellite conjunction analysis and the false confidence theorem', arXiv 5 preprint arXiv:1706.08565.) presents the "false confidence theorem," which has potentially broad implications for statistical inference using Bayesian posterior uncertainty. This theorem says that with an arbitrarily large (sampling/frequentist) probability, there exists a set that does not contain the true parameter value but has an arbitrarily large posterior probability. As the use of Bayesian methods has become increasingly popular in applications of science, engineering and business, it is critically important to understand when Bayesian procedures lead to problematic statistical inferences or interpretations. In this paper, we consider a number of examples demonstrating the paradoxical nature of false confidence to begin to understand the contexts in which the false confidence theorem does (and does not) play a meaningful role in statistical inference. Our examples illustrate that models involving marginalization to non-linear, not one-to-one functions of multiple parameters play a key role in more extreme manifestations of false confidence.

False confidence, non-additive beliefs, and valid statistical inference

“…It is important to emphasize that the theorem only says there exists hypotheses afflicted by false confidence. Certainly, some of these hypotheses are trivial, e.g., complements of very small subsets of Θ, but the examples in Sections 3.1-3.2 reveal that non-trivial hypotheses can be affected too; see, also, Carmichael and Williams (2018). In my experience, the non-trivial yet problematic hypotheses have unusual shapes: the complement of a disc in the plane and the complement of a cone in the upper half-plane in Section 3.1 and 3.2, respectively.…”

Nonpenalized variable selection in high-dimensional linear model settings via generalized fiducial inference

“…For instance, within the Bayesian paradigm, the degree to which a data set provides evidence in support of (or against) an event is quantified by the posterior probability of the event, for any measurable event. Bayesian inference, however, operates by the usual Kolmogorov axioms for probability calculus, and is thereby subject to the false confidence theorem (Balch et al, 2019;Martin, 2019;Carmichael and Williams, 2018), rendering it provably unreliable. The false confidence theorem is mathematical justification for the fact that precise probabilistic-based statistical inferences (e.g., those based on posterior probabilities) are provably unreliable in the sense that there always exists a false hypothesis (with positive Lebesgue measure) having arbitrarily large epistemic (e.g., posterior) probability, with arbitrarily large aleatory (i.e., frequency/frequentist) probability.…”

Discussion of "A Gibbs sampler for a class of random convex polytopes"

“…Since the logic for Bayesian inference is tied to the estimation of a posterior probability distribution, this problem is foundational for Bayesian theory. In my own work, namely Carmichael & Williams (2018), we provide an illustration of simple examples where/how the false confidence theorem manifests.…”