Comparing error minimized extreme learning machines and support vector sequential feed-forward neural networks

Abstract

Recently, error minimized extreme learning machines (EM-ELMs) have been proposed as a simple and efficient approach to build single-hidden-layer feedforward networks (SLFNs) sequentially. They add random hidden nodes one by one (or group by group) and update the output weights incrementally to minimize the sum-of-squares error in the training set. Other very similar methods that also construct SLFNs sequentially had been reported earlier with the main difference that their hidden-layer weights are a subset of the data instead of being random. By analogy with the concept of support vectors original of support vector machines (SVMs), these approaches can be referred to as support vector sequential feed-forward neural networks (SV-SFNNs), and they are a particular case of the Sequential Approximation with Optimal Coefficients and Interacting Frequencies (SAOCIF) method. In this paper, it is firstly shown that EM-ELMs can also be cast as a particular case of SAOCIF. In particular, EM-ELMs can easily be extended to test some number of random candidates at each step and select the best of them, as SAOCIF does. Moreover, it is demonstrated that the cost of the calculation of the optimal output-layer weights in the originally proposed EM-ELMs can be improved if it is replaced by the one included in SAOCIF. Secondly, we present the results of an experimental study on 10 benchmark classification and 10 benchmark regression data sets, comparing EM-ELMs and SV-SFNNs, that was carried out under the same conditions for the two models. Although both models have the same (efficient) computational cost, a statistically significant improvement in generalization performance of SV-SFNNs vs. EM-ELMs was found in 12 out of the 20 benchmark problems.

Benchmarking the selection of the hidden-layer weights in extreme learning machines

“…Algorithm 2 is mainly motivated by the work in [23], where EM-ELMs [16] is compared with SV-SFNNs [17]. EMELMs is a constructive version of ELM which results in a faster method with similar generalization performance.…”

“…The main conclusion in [23] is that SV-SFNNs outperform EM-ELMs, indicating that the strategy of selecting the hiddenlayer weights as a subset of the input data may be better than the random selection made by EM-ELMs. The experiments in this work are devoted to see if that conclusion also holds for the original ELMs.…”

“…SVSFNNs are compared with SVMs in [17], obtaining similar accuracies than SVMs with simpler models. In [23], EM-ELMs are compared with SV-SFNNs, concluding that selecting the hidden-layer weights as a subset of the input data (i.e., with the Input strategy) outperforms the random selection made by EM-ELMs, with the same computational cost.…”

“…The superior performance of SV-SFNNs over EMELMs in [23] may be caused not only by the Input strategy, but by the combination of the Input strategy with a constructive scheme. Except for the preliminary results in [7], to the best of our knowledge it is unknown whether the Input strategy is also able to improve the Random selection of hidden-layer weights in non-constructive ELMs or not, which is the aim of this paper.…”

Extreme Maximum Margin Clustering

“…It should also be noticed that approximation error is not at all guaranteed to be close to zero for every randomly chosen set of hidden nodes. Recently, Romero [20] showed that support vector sequential feedforward neural networks have better generalization performance than error minimized extreme learning machines, which build single-hidden-layer feedforward networks sequentially. Fortunately, the relatively fast con-vergence rate and small approximation error can be guaranteed if the number of hidden nodes is large enough, which is meaningful to large-scale data sets.…”