Information criteria and cross validation for Bayesian inference in regular and singular cases

Abstract

In data science, an unknown information source is estimated by a predictive distribution defined from a statistical model and a prior. In an older Bayesian framework, it was explained that the Bayesian predictive distribution should be the best on the assumption that a statistical model is convinced to be correct and a prior is given by a subjective belief in a small world. However, such a restricted treatment of Bayesian inference cannot be applied to highly complicated statistical models and learning machines in a large world. In 1980, a new scientific paradigm of Bayesian inference was proposed by Akaike, in which both a model and a prior are candidate systems and they had better be designed by mathematical procedures so that the predictive distribution is the better approximation of unknown information source. Nowadays, Akaike’s proposal is widely accepted in statistics, data science, and machine learning. In this paper, in order to establish a mathematical foundation for developing a measure of a statistical model and a prior, we show the relation among the generalization loss, the information criteria, and the cross-validation loss, then compare them from three different points of view. First, their performances are compared in singular problems where the posterior distribution is far from any normal distribution. Second, they are studied in the case when a leverage sample point is contained in data. And last, their stochastic properties are clarified when they are used for the prior optimization problem. The mathematical and experimental comparison shows the equivalence and the difference among them, which we expect useful in practical applications.

State‐space models for ecological time‐series data: Practical model‐fitting

“…A common model selection approach for ecological models is to use information criterion, notably Akaike information criterion (AIC) and its variants (including the method of Bengtsson and Cavanaugh (2006) for SSMs, although this is relatively complex and computationally demanding) within classical analyses. The Watanabe-Akaike information criterion (WAIC; Watanabe, 2013Watanabe, , 2021Watanabe & Opper, 2010) is the preferred criterion for Bayesian analyses. Alternatively, within the Bayesian framework, posterior model probabilities (and the related idea of Evidence; Finke et al (2019)) may be calculated to provide a quantitative comparison of competing models, and permit model-averaged estimates.…”

Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing

“…Eq. ( 4.57) indicates the proportionality between the response of the learning error and the variance of the estimated parameter at equilibrium, which can be regarded as a fluctuation-response relationship described by GAMP [20].…”

Recent advances in algebraic geometry and Bayesian statistics

“…When a leverage sample point is contained in a sample, the importance sampling cross validation Eq. ( 20) becomes unstable [13,33], and the difference between LOOCV and WAIC becomes larger [52,55]. The improved version of numerical calculation of the cross validation was proposed in [39].…”

“…For example, in regression problems where the conditional probability distribution q(y|x) of an output Y for a given input X , the input samples {X i } may dependent or fixed and {Y i } are conditionally independent. In such cases, LOOCV does not estimate the conditional generalization loss, whereas WAIC does [52,55].…”