Mathematical Theory of Bayesian Statistics for Unknown Information Source

Abstract

In statistical inference, uncertainty is unknown and all models are wrong. A person who makes a statistical model and a prior distribution is simultaneously aware that they are fictional and virtual candidates. In order to study such cases, several statistical measures have been constructed, such as cross validation, information criteria, and marginal likelihood, however, their mathematical properties have not yet been completely clarified when statistical models are under-and over-parametrized.In this paper, we introduce a place of mathematical theory of Bayesian statistics for unknown uncertainty, on which we show general properties of cross validation, information criteria, and marginal likelihood. The derived theory holds even if an unknown uncertainty is unrealizable by a statistical model or even if the posterior distribution cannot be approximated by any normal distribution, hence it gives a helpful standpoint for a person who cannot believe in any specific model and prior.The results are followings. (1) There exists a more precise statistical measure of the generalization loss than leave-one-out cross validation and information criterion based on the mathematical properties of them. (2) There exists a more efficient approximation method of the free energy, which is the minus log marginal likelihood, even if the posterior distribution cannot be approximated by any normal distribution. (3) And the prior distributions optimized by the cross validation and the widely applicable information criterion are asymptotically equivalent to each other, which are different from that by the marginal likelihood.

Recent advances in algebraic geometry and Bayesian statistics

“…where Q(q) is a probability distribution on the set of all probability distributions on R N and q(x) is a probability density function which is subject to Q(q). The general pair Q(q) and q(x) contains a specific pair π(θ) and p(x|θ), hence if a person makes a candidate pair π(θ) and p(x|θ) and rejects the existence of unknown Q(q) and q(x), it is a mathematical contradiction [56,57].…”

“…A person cannot believe in a specific pair of a statistical model and a prior distribution because it is under-or over-parametrized in an environment of unknown uncertainty [10,16,17,28]. In other words, a person who makes p(x|θ) and π(θ) is aware that both are only fictional candidates, resulting that it is necessary to check or evaluate their appropriateness from a mathematically general viewpoint [1,2,57]. In an older Bayesian statistics, a person needs to believe that Q(q) = π(θ) and q(x) = p(x|θ), whereas in modern Bayesian statistics, a person is aware to distinguish the unknown data-generating process from a model and a prior [57].…”

“…holds, however, the pair ( p(x|θ ), π(θ )) that minimizes F n is different from the pair that minimizes G n [57]. Minimizing the free energy, which is equivalent to maximizing the marginal likelihood, is often employed in Bayesian model selection and hyperparameter optimization [2].…”

“…Although the cross validation and the information criterion are useful in many statistical applications, these properties clarified a disadvantages of them. Improved methods, adjusted cross validation and information criteria, combining leave-oneout and hold-out cross validations have been proposed [57], which make the variance of the estimators smaller.…”