Generalization Error of Linear Neural Networks in Unidentifiable Cases

Abstract

Abstract. The statistical asymptotic theory is often used in theoretical results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator (MLE) as an normal distribution. However, in layered models such as neural networks, the regularity condition of the asymptotic theory is not necessarily satisfied. The true parameter is not identifiable, if the target function can be realized by a network of smaller size than the size of the model. There has been little known on the behavior of the MLE in these cases of neural networks. In this paper, we analyze the expectation of the generalization error of three-layer linear neural networks, and elucidate a strange behavior in unidentifiable cases. We show that the expectation of the generalization error in the unidentifiable cases is larger than what is given by the usual asymptotic theory, and dependent on the rank of the target function.

Learning efficiency of redundant neural networks in Bayesian estimation

“…However, the asymptotic form of the generalization error has not been clarified because layered neural networks are non-identifiable statistical models [11] [22] [9] [7] [26]. In fact, if a neural network is larger than necessary to attain the true distribution, then the set of true parameters is not one point but an analytic set with singularities, hence neither the distribution of the maximum likelihood estimator nor the Bayesian a posteriori probability density function converges to the normal distribution, even if the number of training samples tends to infinity [8] [24]. Therefore, we can not apply learning theory of regular statistical models to analysis and design of neural networks.…”

Nonparametric Regularization of Decision Trees

“…Compared to [18], our analysis is considerably simpler and, most importantly, covers greedy learners. An actual case analysis for Naive Bayesian classifiers that is guided by a similar idea has been presented by Langley and Sage [10], an actual case analysis for linear neural networks is given in [6].…”

Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance

“…Figures 10 to 13 show the generalization and the training coefficients on the identical conditions to Figures 3 to 6, respectively. The lines in the positive region correspond to the generalization coefficient of the VB approach, clarified in this letter, to that of the ML estimation, clarified in (Fukumizu, 1999), to that of the Bayes estimation, clarified in Aoyagi and Watanabe (2004) 4 and to that of the regular models, respectively; the lines in the negative region correspond to the training coefficient of the VB approach, to that of the ML estimation, and to that of the regular models, respectively. 5 Unfortunately the Bayes training error has not been clarified yet.…”

“…We say that U is the general diagonalized matrix of an N × M matrix T if T has the following singular value decomposition: Although the second term of equation 7.7 is not a simple function, it can relatively easily be numerically calculated by creating samples subject to the Wishart distribution. Furthermore, the simpler function approximating the term can be derived in the large-scale limit when M, N, H, and H * go to infinity in the same order, in a similar fashion to the analysis of the ML estimation (Fukumizu, 1999). We define the following scalars: …”

“…It has thus been known that the ML estimation in general provides poor generalization performance, and in the worst cases, the ML estimator diverges. In linear neural networks, on which we focus in this letter, the generalization error was clarified and proved to be greater than that of the regular models whose dimension of the parameter space is the same when the model is redundant to learn the true distribution (Fukumizu, 1999), although the ML estimator is in a finite region (Baldi & Hornik, 1995).…”