On the problem in model selection of neural network regression in overrealizable scenario

Hagiwara, Katsuyuki; Kuno, Kazuhiro; Usui, Shiro

doi:10.1109/ijcnn.2000.859438

Cited by 11 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We may also consider using likelihood-based model selection criteria, AIC or BIC, which have been widely used for various types of selection problems. However, derivations of these criteria are invalid for mixture models because of the identification problem (Hagiwara et al, 2000;Miloslavsky and Van der Laan, 2003). Nevertheless, because of the computational simplicity, they are still considered as popular model selection criteria for the selection of the number of groups (Miloslavsky and Van der Laan, 2003).…”

Section: Model Comparisons All Competing Models Inmentioning

confidence: 99%

Logistic mixture of multivariate regressions for analysis of water quality impacted by agrochemicals

et al. 2006

View full text Add to dashboard Cite

SUMMARYIn this paper, we study the impacts of two representative agricultural activities, fertilizers and lime application, on water quality. Because of heavy usage of nitrogen fertilizers, nitrate (NO À 3 ) concentration in water is considered as one of the best indicators for agricultural pollution. The mixture of normal distributions has been widely applied with (NO À 3 ) concentrations to cluster water samples into two environmentally interested groups (water impacted by agrochemicals and natural background water groups). However, this method fails to yield satisfying results because it cannot distinguish low-level fertilizer impact and natural background noise. To improve performance of cluster analysis, we introduce the logistic mixture of multivariate regressions model (LMMR). In this approach, water samples are clustered based on the relationships between major element concentrations and physicochemical variables, which are different in impacted water and natural background water.

show abstract

Section: Model Comparisons All Competing Models Inmentioning

confidence: 99%

Logistic mixture of multivariate regressions for analysis of water quality impacted by agrochemicals

et al. 2006

View full text Add to dashboard Cite

show abstract

“…Hartigan has suggested by a sophisticated technique that the likelihood ratio is not of the order of 1/n but of the order of log log n/n in the case of the Gaussian mixture model [13]. Hagiwara's discussion is also a product of elaborate work [12]. Dacunha-Castelle and Gassiat [8] have developed the general framework of this problem.…”

Section: What Happens In Singular Modelsmentioning

confidence: 99%

“…Hagiwara and colleagues have performed simulations and shown that the AIC does not work on neural network models called multilayer perceptrons; they insist that this is caused by a hierarchical property of the model. Using simple models, they have shown that the least square error of the estimator does not obey asymptotically the usual rule of 1/n (n is the number of data) but instead the law of log n/n [12]. Kitahara and colleagues have obtained similar results for different models [15].…”

Section: Introductionmentioning

confidence: 99%

Learning and inference in hierarchical models with singularities

Амари

Ozeki

Park

2003

Systems & Computers in Japan

View full text Add to dashboard Cite

SUMMARYWhen we infer the underlying rule which generates a large amount of data, we assume a family of hierarchical statistical models and estimate an appropriate model and its parameters. In this case, the parameter space of the model usually includes singularities, and interesting phenomena, different from those appearing in conventional inference theory, are observed. In this paper, we review the studies of singular models in learning and inference which are being extensively developed in Japan, and elucidate the mechanisms of strange behavior by using simple models.

show abstract

“…In such learning machines, the map taking parameters to probability distributions is not one-to-one and the Fisher information matrices are singular, hence they are called singular learning machines. For example, three-layered neural networks, normal mixtures, hidden Markov models, Bayesian networks, and reduced rank regressions are singular learning machines [1,2,4,5,6,10]. If a statistical model is singular, then either the maximum likelihood estimator is not subject to the normal distribution even asymptotically or the Bayes posterior distribution can not be approximated by any normal distribution.…”

Section: Introductionmentioning

confidence: 99%

Statistical Learning Theory of Quasi-Regular Cases

Yamada

Watanabe

2012

IEICE Trans. Fundamentals

View full text Add to dashboard Cite

Many learning machines such as normal mixtures and layered neural networks are not regular but singular statistical models, because the map from a parameter to a probability distribution is not one-to-one. The conventional statistical asymptotic theory can not be applied to such learning machines because the likelihood function can not be approximated by any normal distribution. Recently, new statistical theory has been established based on algebraic geometry and it was clarified that the generalization and training errors are determined by two birational invariants, the real log canonical threshold and the singular fluctuation. However, their concrete values are left unknown. In the present paper, we propose a new concept, a quasi-regular case in statistical learning theory. A quasi-regular case is not a regular case but a singular case, however, it has the same property as a regular case. In fact, we prove that, in a quasi-regular case, two birational invariants are equal to each other, resulting that the symmetry of the generalization and training errors holds. Moreover, the concrete values of two birational invariants are explicitly obtained, the quasi-regular case is useful to study statistical learning theory.

show abstract

On the problem in model selection of neural network regression in overrealizable scenario

Cited by 11 publications

References 22 publications

Logistic mixture of multivariate regressions for analysis of water quality impacted by agrochemicals

Logistic mixture of multivariate regressions for analysis of water quality impacted by agrochemicals

Learning and inference in hierarchical models with singularities

Statistical Learning Theory of Quasi-Regular Cases

Contact Info

Product

Resources

About