Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization

Hayashi, Naoki; Watanabe, Sumio

doi:10.1007/s42979-020-0071-3

Cited by 4 publications

(3 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concrete values of them were found in the matrix factorization by Aoyagi and Watanabe (2005), the Poisson mixture by Sato and Watanabe (2019), the latent Dirichlet allocation by Hayashi (2021), and many statistical models and learning machines such as Yamazaki and Watanabe (2003); Yamazaki (2016); Yamazaki and Kaji (2013); Zwiernik (2011); Watanabe (2009). They are useful in the design of Markov chain Monte Calro by Nagata and Watanabe (2008) and the singular Bayesian information criterion by Drton and Plummer (2017).…”

Section: Asymptotic Generalization Lossmentioning

confidence: 99%

Information criteria and cross validation for Bayesian inference in regular and singular cases

Watanabe

2021

Jpn J Stat Data Sci

Self Cite

View full text Add to dashboard Cite

In data science, an unknown information source is estimated by a predictive distribution defined from a statistical model and a prior. In an older Bayesian framework, it was explained that the Bayesian predictive distribution should be the best on the assumption that a statistical model is convinced to be correct and a prior is given by a subjective belief in a small world. However, such a restricted treatment of Bayesian inference cannot be applied to highly complicated statistical models and learning machines in a large world. In 1980, a new scientific paradigm of Bayesian inference was proposed by Akaike, in which both a model and a prior are candidate systems and they had better be designed by mathematical procedures so that the predictive distribution is the better approximation of unknown information source. Nowadays, Akaike’s proposal is widely accepted in statistics, data science, and machine learning. In this paper, in order to establish a mathematical foundation for developing a measure of a statistical model and a prior, we show the relation among the generalization loss, the information criteria, and the cross-validation loss, then compare them from three different points of view. First, their performances are compared in singular problems where the posterior distribution is far from any normal distribution. Second, they are studied in the case when a leverage sample point is contained in data. And last, their stochastic properties are clarified when they are used for the prior optimization problem. The mathematical and experimental comparison shows the equivalence and the difference among them, which we expect useful in practical applications.

show abstract

Section: Asymptotic Generalization Lossmentioning

confidence: 99%

Information criteria and cross validation for Bayesian inference in regular and singular cases

Watanabe

2021

Jpn J Stat Data Sci

Self Cite

View full text Add to dashboard Cite

show abstract

“…According to our prior researches [9,7], several distributions satisfy that the RLCT of NMF is equal to the absolute of the maximum pole of the following zeta function Specifically, when elements of the data matrix follow normal distribution, Poisson distribution, exponential distribution, and Bernoulli distribution, the behavior of the free energy and the generalization error can be describe using RLCT defined by the zeta function . In these previous studies, the prior distribution was limited to positive and bounded, but when proving that the true distribution and the KL information amount of the probabilistic model have the same RLCT as the square norm error of the matrix, the prior distribution is arbitrary.…”

Section: Robustness On Probability Distributionsmentioning

confidence: 99%

Variational approximation error in non-negative matrix factorization

Hayashi

2020

Neural Networks

Self Cite

View full text Add to dashboard Cite

Non-negative matrix factorization (NMF) is a knowledge discovery method that is used for many fields, besides, its variational inference and Gibbs sampling method are also well-known. However, the variational approximation accuracy is not yet clarified, since NMF is not statistically regular and the prior used in the variational Bayesian NMF (VBNMF) has zero or divergence points. In this paper, using algebraic geometrical methods, we theoretically analyze the difference of the negative log evidence (free energy) between VBNMF and Bayesian NMF, and give a lower bound of the approximation accuracy, asymptotically. The results quantitatively show how well the VBNMF algorithm can approximate Bayesian NMF.

show abstract

“…Determining RLCTs is important for estimating the sufficient sample size, constructing procedures of learning, and selecting models. In fact, RLCTs of many singular models have been studied: mixture models [63,65,47,37,60], Boltzmann machines [67,4,5], non-negative matrix factorization [22,21,18], latent class analysis [13], latent Dirichlet allocation [23,19], naive Bayesian networks [46], Bayesian networks [64],…”

Section: Introductionmentioning

confidence: 99%

Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization

Cited by 4 publications

References 45 publications

Information criteria and cross validation for Bayesian inference in regular and singular cases

Information criteria and cross validation for Bayesian inference in regular and singular cases

Variational approximation error in non-negative matrix factorization

The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

Contact Info

Product

Resources

About