How Should We Quantify Uncertainty in Statistical Inference?

Abstract

An inferential statement is any statement about the parameters, form of the underlying process or future outcomes. An inferential statement, that provides an approximation to the truth, becomes "statistical" only when there is a measure of uncertainty associated with it. The uncertainty of an inferential statement is generally quantified in terms of probability of the strength of approximation to the truth. This is what we term "inferential uncertainty." Answer to this question has significant implications in statistical decision making where inferential uncertainty is combined with loss functions for predicted outcomes to compute the risk associated with the decision. The Classical and the Evidential paradigms use aleatory (frequency based) probability for quantifying uncertainty whereas the Bayesian approach utilizes epistemic (belief based) probability. To compute aleatory uncertainty, one needs to answer the question: which experiment is being repeated, hypothetically or otherwise? whereas computing epistemic uncertainty requires: What is the prior belief? Deciding which type of uncertainty is appropriate for scientific inference has been a contentious issue and without proper resolution because it has been commonly formulated in terms of statements about parameters, that are statistical constructs, not observables. Common to these approaches is the desire to understand the data generating mechanism. Whether one follows the Frequentist or the Bayesian approach inferential statements concerning prediction are aleatory in nature and are practically ascertainable. We consider the desirable characteristics for quantification of uncertainty as: (1) Parameterization and data transformation invariance, (2) correct predictive coverage, (3) uncertainty that depends only on the data at hand and the hypothesized data generating mechanism, and (4) diagnostics for model misspecification and guidance for correction. We examine the Classical, Bayesian and Evidential approaches in the light of these characteristics. Unfortunately, none of these inferential approaches possesses all of our desiderata although the Evidential approach seems to come closest. Choosing an inferential approach, thus, involves choosing between either specifying the hypothetical experiment that will be repeated or equivalently a sampling distribution of the estimator or a prior distribution on the model space or an evidence function.

Enhancing Clinical Data Analysis by Explaining Interaction Effects between Covariates in Deep Neural Network Models

“…Many factors influence the final trained model, which include the initial randomly assigned weights, the process of training (e.g., the size and order of the mini-batches), and the regularization method (e.g., early stopping, lasso). This phenomenon is not common in traditional statistical analysis, but has received some attention [ 27 , 28 ]. We plan to investigate these uncertainty issues in the context of DNN models in the future.…”

Grand challenges in mathematical biology: Integrating multi-scale modeling and data

“…As an example we mention the molecular-level data collected via 30], their use opens up new challenges in understanding the huge amount of generated data: from new approaches to deal with high data volumes generated at higher and higher speeds and that could be presented in a variety of forms (structured, semi-structured, and/or unstructured data) [24], to new approaches to deal with data heterogeneity [22,24], or deal with incomplete data [21,24] or even irreproducible data-which is a major issue at least in immunology and cell biology [31,32], and even challenges in understanding the biological mechanisms behind the data [33]. While artificial intelligence techniques (e.g., machine learning, natural language processing, computational intelligence) can provide faster and more accurate results in data analytics compared to classical statistical methods [24] (especially if the training data is not biased in any way) they don't provide us with a mechanistic understanding of the data.…”

Assessing the Global and Local Uncertainty of Scientific Evidence in the Presence of Model Misspecification

“…If the data are highly informative either by design or by chance, we should be quite confident about our estimate of the total population size, irrespective of what other experimenters might observe. It can be shown (see review in Lele, 2020b) that prediction of a new observation based on local uncertainty is more accurate than prediction based on global uncertainty. However, this result also depends on correct model specification.…”