The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

Hayashi, Naoki

doi:10.1016/j.neunet.2021.01.024

Cited by 11 publications

(8 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The generalization loss G n (π), the leave-one-out cross validation C n (π), the widely applicable information criterion W n (π), the hold-out cross validation H n (π) , and the adjusted cross validation A n (π) can also be defined by the same eqs. ( 11), ( 16), (17). (22), and (23), respectively.…”

Section: Evaluation Of Prior Distributionsmentioning

confidence: 99%

“…The concrete values of RLCTs of singular statistical models have been clarified, for example, reduced rank regressions [4], neural networks [5,31], normal mixtures [44], Poisson mixtures [24], Latent Dirichlet allocations [17], and multinomial mixtures [42]. It is also clarified that, based on RLCT, the exchange probability of exchange MCMC methods can be optimally designed [20].…”

Section: Real Log Canonical Thresholdmentioning

confidence: 99%

See 1 more Smart Citation

Mathematical Theory of Bayesian Statistics for Unknown Information Source

Watanabe¹

2022

Preprint

View full text Add to dashboard Cite

In statistical inference, uncertainty is unknown and all models are wrong. A person who makes a statistical model and a prior distribution is simultaneously aware that they are fictional and virtual candidates. In order to study such cases, several statistical measures have been constructed, such as cross validation, information criteria, and marginal likelihood, however, their mathematical properties have not yet been completely clarified when statistical models are under-and over-parametrized.In this paper, we introduce a place of mathematical theory of Bayesian statistics for unknown uncertainty, on which we show general properties of cross validation, information criteria, and marginal likelihood. The derived theory holds even if an unknown uncertainty is unrealizable by a statistical model or even if the posterior distribution cannot be approximated by any normal distribution, hence it gives a helpful standpoint for a person who cannot believe in any specific model and prior.The results are followings. (1) There exists a more precise statistical measure of the generalization loss than leave-one-out cross validation and information criterion based on the mathematical properties of them. (2) There exists a more efficient approximation method of the free energy, which is the minus log marginal likelihood, even if the posterior distribution cannot be approximated by any normal distribution. (3) And the prior distributions optimized by the cross validation and the widely applicable information criterion are asymptotically equivalent to each other, which are different from that by the marginal likelihood.

show abstract

Section: Evaluation Of Prior Distributionsmentioning

confidence: 99%

Section: Real Log Canonical Thresholdmentioning

confidence: 99%

Mathematical Theory of Bayesian Statistics for Unknown Information Source

Watanabe¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These properties are helpful to study RCLTs of statistical models and learning machines. In fact, RLCTs of important statistical models and learning machines were found by developing resolution procedures in neural networks [8,44], normal mixtures [60], Poisson mixtures [35], multinomial mixtures [58], general and nonnegative matrix facorizations [6,21], Boltzmann machines [7,62], hidden and general Markov models [61,67] and latent Dirichlet allocations [22]. Note that singularities in statistical models and learning machines make the free energy and the generalization loss smaller if Bayesian inferences are employed, hence almost all learning machines are singular [47] and that's good [59].…”

Section: Assume That λmentioning

confidence: 99%

“…Such nonidentifiable and singular models and machines are not special but ubiquitous in modern statistics and machine learning. For example, neural networks and deep learning [8,43,44] have hierarchical structures, normal mixtures [24,60], Poisson mixtures [35], multinomial mixtures [58], and latent Dirichlet allocations [22] have latent or hidden variables, and matrix factorizations [6,21], Boltzmann machines [7,62], and Markov models [61,67] have both hidden and hierarchical parts. In other words, almost all statistical models and learning machines which extract hidden structures or hierarchical inferences are nonidentifiable and singular [47].…”

Section: Introductionmentioning

confidence: 99%

Recent advances in algebraic geometry and Bayesian statistics

Watanabe

2022

Info. Geo.

View full text Add to dashboard Cite

This article is a review of theoretical advances in the research field of algebraic geometry and Bayesian statistics in the last two decades. Many statistical models and learning machines which contain hierarchical structures or latent variables are called nonidentifiable, because the map from a parameter to a statistical model is not one-to-one. In nonidentifiable models, both the likelihood function and the posterior distribution have singularities in general, hence it was difficult to analyze their statistical properties. However, from the end of the 20th century, new theory and methodology based on algebraic geometry have been established which enable us to investigate such models and machines in the real world. In this article, the following results in recent advances are reported. First, we explain the framework of Bayesian statistics and introduce a new perspective from the birational geometry. Second, two mathematical solutions are derived based on algebraic geometry. An appropriate parameter space can be found by a resolution map, which makes the posterior distribution be normal crossing and the log likelihood ratio function be well-defined. Third, three applications to statistics are introduced. The posterior distribution is represented by the renormalized form, the asymptotic free energy is derived, and the universal formula among the generalization loss, the cross validation, and the information criterion is established. Two mathematical solutions and three applications to statistics based on algebraic geometry reported in this article are now being used in many practical fields in data science and artificial intelligence. KeywordsBirational geometry • Resolution of singularities • Bayesian statistics • Real log canonical threshold Communicated by Noboru Murata.

show abstract

“…In general, the parameter set K(w) = 0 contains complicated singularities, hence it is difficult to find the resolution map, however, both RLCTs and multiplicities have been clarified in several statistical models and learning machines. Examples of the models in which the RLCTs are found include normal mixtures [10], Poisson mixtures [11], Bernoulli mixtures [12], rank regression [10], Latent Dirichlet Allocation (LDA) [13], and so on. In addition, the RLCTs are used as an analysis of the exchange rate of the replica exchange method [14], which is one of the Markov chain Monte Carlo methods.…”

Section: Introductionmentioning

confidence: 99%

Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures

Watanabe¹,

Watanabe²

2022

Preprint

View full text Add to dashboard Cite

Multinomial mixtures are widely used in the information engineering field, however, their mathematical properties are not yet clarified because they are singular learning models. In fact, the models are non-identifiable and their Fisher information matrices are not positive definite. In recent years, the mathematical foundation of singular statistical models are clarified by using algebraic geometric methods. In this paper, we clarify the real log canonical thresholds and multiplicities of the multinomial mixtures and elucidate their asymptotic behaviors of generalization error and free energy.

show abstract

The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

Cited by 11 publications

References 56 publications

Mathematical Theory of Bayesian Statistics for Unknown Information Source

Mathematical Theory of Bayesian Statistics for Unknown Information Source

Recent advances in algebraic geometry and Bayesian statistics

Asymptotic Behavior of Bayesian Generalization Error in Multinomial Mixtures

Contact Info

Product

Resources

About