On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models

2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)

2008

Almost all learning machines used in computational intelligence are not regular but singular statistical models, because they are nonidentifiable and their Fisher information matrices are singular. In singular learning machines, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies the asymptotic normality, resulting that it has been difficult to estimate generalization performances. In this paper, we establish a formula of equations of states which holds among Bayes and Gibbs generalization and training errors, and show that two generalization errors can be estimated from two training errors. The equations of states proved in this paper hold for any true distribution, any learning machine, and a priori distribution, and any singularities, hence they define widely applicable information criteria.

Section: A Backgroundmentioning

confidence: 99%

A formula of equations of states in singular learning machines

2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)

2008

“…In such learning machines, the map taking parameters to probability distributions is not one-to-one and the Fisher information matrices are singular, hence they are called singular learning machines. For example, three-layered neural networks, normal mixtures, hidden Markov models, Bayesian networks, and reduced rank regressions are singular learning machines [1,2,4,5,6,10]. If a statistical model is singular, then either the maximum likelihood estimator is not subject to the normal distribution even asymptotically or the Bayes posterior distribution can not be approximated by any normal distribution.…”

Section: Introductionmentioning

confidence: 99%

Statistical Learning Theory of Quasi-Regular Cases

Yamada

IEICE Trans. Fundamentals

2012

Many learning machines such as normal mixtures and layered neural networks are not regular but singular statistical models, because the map from a parameter to a probability distribution is not one-to-one. The conventional statistical asymptotic theory can not be applied to such learning machines because the likelihood function can not be approximated by any normal distribution. Recently, new statistical theory has been established based on algebraic geometry and it was clarified that the generalization and training errors are determined by two birational invariants, the real log canonical threshold and the singular fluctuation. However, their concrete values are left unknown. In the present paper, we propose a new concept, a quasi-regular case in statistical learning theory. A quasi-regular case is not a regular case but a singular case, however, it has the same property as a regular case. In fact, we prove that, in a quasi-regular case, two birational invariants are equal to each other, resulting that the symmetry of the generalization and training errors holds. Moreover, the concrete values of two birational invariants are explicitly obtained, the quasi-regular case is useful to study statistical learning theory.

“…For singular learning machines, the log likelihood function can not be approximated by any quadratic form of the parameter, with the result that the conventional relationship between generalization errors and training errors does not hold either for the maximum likelihood method [6] [5] [7] or Bayes estimation [12]. Singularities strongly affect generalization performances [15] and learning dynamics [1].…”

Section: Introductionmentioning

confidence: 99%

Equations of states in singular statistical estimation

2010

Neural Networks

Learning machines that have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither does the Bayes a posteriori distribution converge to the normal distribution nor does the maximum likelihood estimator satisfy asymptotic normality. This is the main reason that it has been difficult to predict their generalization performance from trained states. In this paper, we study four errors, (1) the Bayes generalization error, (2) the Bayes training error, (3) the Gibbs generalization error, and (4) the Gibbs training error, and prove that there are universal mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that the Bayes and Gibbs generalization errors can be estimated by Bayes and Gibbs training errors, and we propose widely applicable information criteria that can be applied to both regular and singular statistical models.