On the problem of applying AIC to determine the structure of a layered feedforward neural network

Hagiwara, Katsuyuki; Toda, Naohiro; Usui, Shiro

doi:10.1109/ijcnn.1993.714176

Cited by 25 publications

(14 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, the matrices F and K are of the order O (1) as N goes to infinity. We write ε = 1 √ N hereafter for notational simplicity, and obtain the expansion of…”

Section: A Proof Of Theoremmentioning

confidence: 99%

See 1 more Smart Citation

Generalization Error of Linear Neural Networks in Unidentifiable Cases

Fukumizu

1999

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The statistical asymptotic theory is often used in theoretical results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator (MLE) as an normal distribution. However, in layered models such as neural networks, the regularity condition of the asymptotic theory is not necessarily satisfied. The true parameter is not identifiable, if the target function can be realized by a network of smaller size than the size of the model. There has been little known on the behavior of the MLE in these cases of neural networks. In this paper, we analyze the expectation of the generalization error of three-layer linear neural networks, and elucidate a strange behavior in unidentifiable cases. We show that the expectation of the generalization error in the unidentifiable cases is larger than what is given by the usual asymptotic theory, and dependent on the rank of the target function.

show abstract

“…Then, the matrices F and K are of the order O (1) as N goes to infinity. We write ε = 1 √ N hereafter for notational simplicity, and obtain the expansion of…”

Section: A Proof Of Theoremmentioning

confidence: 99%

“…It has been clarified recently that the usual statistical asymptotic theory on the MLE does not necessarily hold in neural networks ( [1], [2]). This always happens if we consider the model selection problem in neural networks.…”

Section: Introductionmentioning

confidence: 99%

Generalization Error of Linear Neural Networks in Unidentifiable Cases

Fukumizu

1999

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…According to this property, called asymptotic normality, the well-known criterion AIC (Akaike information criterion; Akaike, 1974) is derived. In the case of MLP, however, we cannot show the asymptotic normality of estimators because of the unidentifiability of optimal parameters, so that the effectiveness of AIC is not ensured (Hagiwara, Toda, & Usui, 1993;Anders & Korn, 1999). Although some other criteria, such as NIC (network information criterion; Murata, Yoshizawa, & Amari, 1994) and GPE (generalized prediction error; Moody, 1992), have been proposed for MLP, they are effective only when the asymptotic normality holds.…”

Section: Introductionmentioning

confidence: 96%

On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models

2004

View full text Add to dashboard Cite

In order to analyze the stochastic property of multilayered perceptrons or other learning machines, we deal with simpler models and derive the asymptotic distribution of the least-squares estimators of their parameters. In the case where a model is unidentified, we show different results from traditional linear models: the well-known property of asymptotic normality never holds for the estimates of redundant parameters.

show abstract

“…Hierarchical learning models, such as the layered neural network, the Boltzmann machine, the reduced rank regression and the normal mixture model, are known to be effective learning models for analyzing such data. These are, however, singular learning models, which cannot be analyzed using the classic theory of regular statistical models, because singular learning models have a singular Fisher metric that is not always approximated by any quadratic form [1][2][3][4]. Therefore, it is difficult to analyze their generalization errors, which indicate how precisely the predictive function approximates the true density function.…”

Section: Introductionmentioning

confidence: 99%

Consideration on Singularities in Learning Theory and the Learning Coefficient

Aoyagi

2013

Entropy

View full text Add to dashboard Cite

Abstract:We consider the learning coefficients in learning theory and give two new methods for obtaining these coefficients in a homogeneous case: a method for finding a deepest singular point and a method to add variables. In application to Vandermonde matrix-type singularities, we show that these methods are effective. The learning coefficient of the generalization error in Bayesian estimation serves to measure the learning efficiency in singular learning models. Mathematically, the learning coefficient corresponds to a real log canonical threshold of singularities for the Kullback functions (relative entropy) in learning theory.

show abstract

On the problem of applying AIC to determine the structure of a layered feedforward neural network

Cited by 25 publications

References 3 publications

Generalization Error of Linear Neural Networks in Unidentifiable Cases

Generalization Error of Linear Neural Networks in Unidentifiable Cases

On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models

Consideration on Singularities in Learning Theory and the Learning Coefficient

Contact Info

Product

Resources

About