Improved learning algorithms for mixture of experts in multiclass classification

Chen, Ke; Xu, Lei; Chi, Huisheng

doi:10.1016/s0893-6080(99)00043-x

Cited by 113 publications

(86 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It can be calculated by Iteratively Reweighted Least Squares (IRLS) (Jacobs et al, 1991;Jordan and Jacobs, 1994;Chen et al, 1999;Green, 1984;Chamroukhi et al, 2009). …”

Section: The Normal Moe (Nmoe) Model and Its Maximum Likelihood Estimmentioning

confidence: 99%

Robust mixture of experts modeling using thetdistribution

Chamroukhi¹

2016

Neural Networks

View full text Add to dashboard Cite

Mixture of Experts (MoE) is a popular framework for modeling heterogeneity in data for regression, classification, and clustering. For regression and cluster analyses of continuous data, MoE usually use normal experts following the Gaussian distribution. However, for a set of data containing a group or groups of observations with heavy tails or atypical observations, the use of normal experts is unsuitable and can unduly affect the fit of the MoE model. We introduce a robust MoE modeling using the t distribution. The proposed t MoE (TMoE) deals with these issues regarding heavy-tailed and noisy data. We develop a dedicated expectation-maximization (EM) algorithm to estimate the parameters of the proposed model by monotonically maximizing the observed data log-likelihood. We describe how the presented model can be used in prediction and in model-based clustering of regression data. The proposed model is validated on numerical experiments carried out on simulated data, which show the effectiveness and the robustness of the proposed model in terms of modeling non-linear regression functions as well as in model-based clustering. Then, it is applied to the real-world data of tone perception for musical data analysis, and the one of temperature anomalies for the analysis of climate change data. The obtained results show the usefulness of the TMoE model for practical applications.

show abstract

“…It can be calculated by Iteratively Reweighted Least Squares (IRLS) (Jacobs et al, 1991;Jordan and Jacobs, 1994;Chen et al, 1999;Green, 1984;Chamroukhi et al, 2009). …”

Section: The Normal Moe (Nmoe) Model and Its Maximum Likelihood Estimmentioning

confidence: 99%

Robust mixture of experts modeling using thetdistribution

Chamroukhi¹

2016

Neural Networks

View full text Add to dashboard Cite

show abstract

“…Each of these, derives a conditional probability density of the target output. The outputs of expert networks are combined using a gating network trained to select the expert(s) that has the best performance in solving the problem [4][5][6][7]. In the basic form of ME [3], the expert and gating networks are linear classifiers, however, for more complex classification tasks, the expert and gating networks could be more complicated.…”

Section: Introductionmentioning

confidence: 99%

Investigation the Effect of Particle Swarm Optimization in Performance of Mixture of Experts

Rani¹,

Hatami²,

Meysamiazad³

2014

IJCA

View full text Add to dashboard Cite

Mixture of experts (ME) is one of the most popular and interesting combining methods, which has great potential to improve performance in machine learning. ME is established based on the divide-and-conquer principle in which the problem space is divided between a few neural network experts, supervised by a gating network. In earlier works on ME, different strategies were developed to divide the problem space between the experts. As result, we have introduced a new method based on the principles of Particle Swarm Optimization (PSO) as a learning step in ME. In this paper, different aspects of the proposed method are compared with the common version of ME. The result carried out from this paper shows that the new method is robust to the variation of ensemble complexity in terms of the number of individual experts, and the number of hidden units.

show abstract

“…These approaches will be described further in Section III. For the ME networks, it is reported in the literature [15] that networks trained by the EM algorithm using the IRLS algorithm in the inner loop of the M-step, often performed poorly in multiclass classification because of the incorrect assumption on parameter independence. In our study, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing even though the assumption of independence is incorrect.…”

Section: Introductionmentioning

confidence: 99%

Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

McLachlan

2004

IEEE Trans. Neural Netw.

View full text Add to dashboard Cite

Abstract-The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.Index Terms-Expectation-conditional maximization (ECM) algorithm, expectation-maximization (EM) algorithm, mixture of experts, multiclass classification, multilayer perceptron (MLP), variational relaxation.

show abstract

Improved learning algorithms for mixture of experts in multiclass classification

Cited by 113 publications

References 22 publications

Robust mixture of experts modeling using thetdistribution

Robust mixture of experts modeling using thetdistribution

Investigation the Effect of Particle Swarm Optimization in Performance of Mixture of Experts

Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

Contact Info

Product

Resources

About