Impact of Variances of Random Weights and Biases on Extreme Learning Machine

Abstract

Abstract:Although the uniform convergence of extreme learning machine (ELM) has been proved for any continues probability distribution, the variances of random numbers initializing input-layer weights and hidden-layer biases indeed have the obvious impact on generalization performance of ELM. In this paper, we validate this effect by testing the classification accuracies of ELMs initialized by the random numbers with different variances. We select three commonly-used probability distributions (i.e., Uniform, Gamma and Normal) and 30 UCI data sets to conduct our comparative study. The experimental results present some important and valuable observations and instructions: (1) Uniform and Gamma distributions with the smaller variances usually make ELMs get the higher training and testing accuracies; (2) In comparison with Normal distribution, the variances of Uniform and Gamma distributions have the significant impact on classification performance of ELMs; (3) Uniform and Gamma distributions with the larger variances could seriously degrade the classification capability of ELMs; (4) ELMs initialized by Uniform and Gamma distributions with the larger variances generally needs the more hidden-layer nodes to achieve the equivalent classification accuracies with ones having the smaller variances; and (5) Normal distribution are more easily lead to the over-fitting of ELMs.

Extreme learning machine versus classical feedforward network

“…Even though ELM parameters are chosen at random, the probability distribution used for sampling them may impact the model performance. Authors of [45] compare various distributions with different variances. Authors recommend an excellent default choice of distribution-Gaussian with mean equal to 0 and a standard deviation less than or equal to 0.1.…”