Stochastic complexities of reduced rank regression in Bayesian estimation

Aoyagi, Miki; Watanabe, Sumio

doi:10.1016/j.neunet.2005.03.014

Cited by 91 publications

(93 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By Proposition 1.1.4 and after some simplification, this probability equals n u 1+ u 1+ u 11 n − u 1+ u +1 − u 11 p u1+ 1+ p n−u1+…”

Section: Hypothesis Tests For Contingencymentioning

confidence: 92%

“…The asymptotics of marginal likelihood integrals associated with this parametrization of reduced rank regression models were studied in a paper by Aoyagi and Watanabe [11]. The problem at the core of this work is to determine the asymptotics of integrals of the form…”

Section: Information Criteria and Asymptoticsmentioning

confidence: 99%

“…Since the preimages g −1 h (θ 0 ) are not always compact, it is assumed in [11] that the prior density p(α, β) has compact support Ω and is positive at (α 0 , β 0 ). Then the learning coefficient and associated multiplicity are derived for arbitrary values of a, b and h. We illustrate here the case of rank h = 1.…”

Section: Information Criteria and Asymptoticsmentioning

confidence: 99%

See 2 more Smart Citations

Lectures on Algebraic Statistics

2009

View full text Add to dashboard Cite

“…By Proposition 1.1.4 and after some simplification, this probability equals n u 1+ u 1+ u 11 n − u 1+ u +1 − u 11 p u1+ 1+ p n−u1+…”

Section: Hypothesis Tests For Contingencymentioning

confidence: 92%

Section: Information Criteria and Asymptoticsmentioning

confidence: 99%

Section: Information Criteria and Asymptoticsmentioning

confidence: 99%

See 1 more Smart Citation

Lectures on Algebraic Statistics

2009

View full text Add to dashboard Cite

“…By the fundamental condition (A.3), M(x) 2 Q(x) is an integrable function, hence S(x, u) is bounded by the integrable function. By using Lebesgue's convergence theorem, as u k → 0, we obtain 1 = a(x, u) 2 2 q(x)dx for any u that satisfies u 2k = 0, which proves eq. …”

Section: Proof Of Lemmamentioning

confidence: 65%

“…Since the constant λ depends strongly on the true distribution, the learning machine, and the a priori distribution, it characterizes the properties of learning machines. The values of several models have been studied in neural networks [16], normal mixtures [24], reduced rank regressions [2], Boltzmann machines [25], and hidden Markov models [26]. Also the behavior of λ was analyzed for the case when Jeffreys' prior is employed as an a priori distribution [14], and in the case when the distance of the true distribution from the singularity is in proportion to 1/ √ n [18].…”

Section: Corollarymentioning

confidence: 99%

Equations of states in singular statistical estimation

Watanabe

2010

Neural Networks

Self Cite

View full text Add to dashboard Cite

Learning machines that have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither does the Bayes a posteriori distribution converge to the normal distribution nor does the maximum likelihood estimator satisfy asymptotic normality. This is the main reason that it has been difficult to predict their generalization performance from trained states. In this paper, we study four errors, (1) the Bayes generalization error, (2) the Bayes training error, (3) the Gibbs generalization error, and (4) the Gibbs training error, and prove that there are universal mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that the Bayes and Gibbs generalization errors can be estimated by Bayes and Gibbs training errors, and we propose widely applicable information criteria that can be applied to both regular and singular statistical models.

show abstract

Stochastic complexity of complete bipartite graph‐type Boltzmann machines in mean field approximation

Nishiyama

Watanabe

2007

Electron Comm Jpn Pt III

Self Cite

View full text Add to dashboard Cite

SUMMARYA variational Bayes method is proposed as an approximation method for implementing Bayesian posterior distribution with moderate computational complexity. Its effectiveness for real problems has been confirmed. The variational Bayes method is a method which generalizes the mean field approximation used in calculating partition functions in statistical physics. In recent years, the mathematical properties of the precision with which it is approximated have been investigated. In this paper, the authors consider the asymptotic stochastic complexity in the case of applying the mean field approximation to complete bipartite graph-type Boltzmann machines and theoretically derive that asymptotic form. Also, based on the results, the authors quantitatively consider the difference between Bayesian posterior distribution and the posterior distribution of the mean field approximation. © 2007 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 90(9): 1-9, 2007; Published online in Wiley InterScience (www.interscience. wiley.com).

show abstract

Stochastic complexities of reduced rank regression in Bayesian estimation

Cited by 91 publications

References 11 publications

Lectures on Algebraic Statistics

Lectures on Algebraic Statistics

Equations of states in singular statistical estimation

Stochastic complexity of complete bipartite graph‐type Boltzmann machines in mean field approximation

Contact Info

Product

Resources

About