False confidence, non-additive beliefs, and valid statistical inference

Abstract

Statistics has made tremendous advances since the times of Fisher, Neyman, Jeffreys, and others, but the fundamental questions about probability and inference that puzzled our founding fathers still exist and might even be more relevant today. To overcome these challenges, I propose to look beyond the two dominating schools of thought and ask what do scientists need out of statistics, do the existing frameworks meet these needs, and, if not, how to fill the void? To the first question, I contend that scientists seek to convert their data, posited statistical model, etc., into calibrated degrees of belief about quantities of interest. To the second question, I argue that any framework that returns additive beliefs, i.e., probabilities, necessarily suffers from false confidence-certain false hypotheses tend to be assigned high probability-and, therefore, risks making systematically misleading conclusions. This reveals the fundamental importance of non-additive beliefs in the context of statistical inference. But non-additivity alone is not enough so, to the third question, I offer a sufficient condition, called validity, for avoiding false confidence, and present a framework, based on random sets and belief functions, that provably meets this condition.

Response to the comment Confidence in confidence distributions!

“…4.3) based on an alternative approach, the so-called inferential model (IM) framework, which works directly in the domain of imprecise probability through the use of random sets. To our knowledge, the only general approach to distributional inference without false confidence is the IM framework; see Martin & Liu [24,25] and Martin [26].…”

A Comparison of Learning Rate Selection Methods in Generalized Bayesian Inference

“…In many cases, it is more natural to formulate the inference problem with a loss function rather than a statistical model. These are often referred to as Gibbs posterior distributions; see Syring and Martin (2017, 2019, 2020b, Bhattacharya and Martin (2022), Wang and Martin (2020), and Section 6 below.…”

“…Real coverage probabilities differing significantly from advertised levels is a serious concern (Fraser, 2011;Martin, 2019). A gap between real and advertised coverage probabilities can have various causes, but here we focus on model misspecification.…”

Satellite conjunction analysis and the false confidence theorem

“…Risk analysis problems with bounded failure domains are strong candidates for severe false confidence. Recent work demonstrates that false confidence can also arise as the result of nonlinear uncertainty propagation, even if a marginalization-specific reference posterior is used [54,55].…”