Asymptotic analysis of Bayesian generalization error with Newton diagram

Yamazaki, Keisuke; Aoyagi, Miki; Watanabe, Sumio

doi:10.1016/j.neunet.2009.07.029

Cited by 43 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For several statistical models, the coefficient λ or its upper bound was evaluated by analyzing the pole of the zeta function (Yamazaki and Watanabe 2003a, 2003bRusakov and Geiger 2005;Aoyagi and Watanabe 2005;Watanabe 2009;Yamazaki et al 2010). The condition that the true distribution is contained in the model is natural and essential for dealing with model selection problems, which is in fact assumed in these analyses (Schwarz 1978;Yamazaki and Watanabe 2003a, 2003bRusakov and Geiger 2005;Aoyagi and Watanabe 2005;Yamazaki et al 2010).…”

Section: Bayesian Learningmentioning

confidence: 99%

“…The condition that the true distribution is contained in the model is natural and essential for dealing with model selection problems, which is in fact assumed in these analyses (Schwarz 1978;Yamazaki and Watanabe 2003a, 2003bRusakov and Geiger 2005;Aoyagi and Watanabe 2005;Yamazaki et al 2010).…”

Section: Bayesian Learningmentioning

confidence: 99%

“…For several statistical models, tighter bounds or exact evaluation of the coefficient λ of the free energy (9) has been obtained recently (Aoyagi and Watanabe 2005;Yamazaki et al 2010). If the free energy and the minimum variational free energy have the asymptotic forms, F * (n) λ log n + o(log n) and F * min (n) λ log n + o(log n) respectively and λ < λ, the approximation accuracy of the variational Bayes approach is evaluated as…”

Section: Accuracy Of Approximationmentioning

confidence: 99%

See 2 more Smart Citations

An alternative view of variational Bayes and asymptotic approximations of free energy

2011

View full text Add to dashboard Cite

Bayesian learning, widely used in many applied data-modeling problems, is often accomplished with approximation schemes because it requires intractable computation of the posterior distributions. In this study, we focus on two approximation methods, variational Bayes and local variational approximation. We show that the variational Bayes approach for statistical models with latent variables can be viewed as a special case of local variational approximation, where the log-sum-exp function is used to form the lower bound of the log-likelihood. The minimum variational free energy, the objective function of variational Bayes, is analyzed and related to the asymptotic theory of Bayesian learning. This analysis additionally implies a relationship between the generalization performance of the variational Bayes approach and the minimum variational free energy.

show abstract

Section: Bayesian Learningmentioning

confidence: 99%

Section: Bayesian Learningmentioning

confidence: 99%

Section: Accuracy Of Approximationmentioning

confidence: 99%

See 1 more Smart Citation

An alternative view of variational Bayes and asymptotic approximations of free energy

2011

View full text Add to dashboard Cite

show abstract

“…The singularities in the parameter space play an important role to determine the performance. Mathematical approaches to reveal the structures of singularities have been developed [7], [8]. Based on the algebraic geometrical method, singularities in many models have been analyzed [9]- [17].…”

Section: Introductionmentioning

confidence: 99%

Asymptotic Marginal Likelihood on Linear Dynamical Systems

Naito

Yamazaki

2014

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

SUMMARYLinear dynamical systems are basic state space models literally dealing with underlying system dynamics on the basis of linear state space equations. When the model is employed for time-series data analysis, the system identification, which detects the dimension of hidden state variables, is one of the most important tasks. Recently, it has been found that the model has singularities in the parameter space, which implies that analysis for adverse effects of the singularities is necessary for precise identification. However, the singularities in the models have not been thoroughly studied. There is a previous work, which dealt with the simplest case; the hidden state and the observation variables are both one dimensional. The present paper extends the setting to general dimensions and more rigorously reveals the structure of singularities. The results provide the asymptotic forms of the generalization error and the marginal likelihood, which are often used as criteria for the system identification.

show abstract

“…The reason why we contribute only to such singularities is that the Vandermonde matrix type is generic and essential in learning theory. These log canonical thresholds give the learning coefficients of normal mixture models, three-layered neural networks and mixtures of binomial distributions, which are widely used as effective learning models (Sections 3.1 and 3.2 and [13]). Moreover, we prove Theorem 2 (the method for finding a deepest deepest singular point) and Theorem 3 (the method to add variables), which are very beneficial to obtain the log canonical threshold for the homogeneous case.…”

Section: Introductionmentioning

confidence: 99%

Consideration on Singularities in Learning Theory and the Learning Coefficient

Aoyagi

2013

Entropy

Self Cite

View full text Add to dashboard Cite

Abstract:We consider the learning coefficients in learning theory and give two new methods for obtaining these coefficients in a homogeneous case: a method for finding a deepest singular point and a method to add variables. In application to Vandermonde matrix-type singularities, we show that these methods are effective. The learning coefficient of the generalization error in Bayesian estimation serves to measure the learning efficiency in singular learning models. Mathematically, the learning coefficient corresponds to a real log canonical threshold of singularities for the Kullback functions (relative entropy) in learning theory.

show abstract

Asymptotic analysis of Bayesian generalization error with Newton diagram

Cited by 43 publications

References 17 publications

An alternative view of variational Bayes and asymptotic approximations of free energy

An alternative view of variational Bayes and asymptotic approximations of free energy

Asymptotic Marginal Likelihood on Linear Dynamical Systems

Consideration on Singularities in Learning Theory and the Learning Coefficient

Contact Info

Product

Resources

About