Big data regression with parallel enhanced and convex incremental extreme learning machines

Abstract

This work considers scalable incremental extreme learning machine (I-ELM) algorithms, which could be suitable for big data regression. During the training of I-ELMs, the hidden neurons are presented one by one, and the weights are based solely on simple direct summations, which can be most efficiently mapped on parallel environments. Existing incremental versions of ELMs are the I-ELM, enhanced incremental ELM (EI-ELM), and convex incremental ELM (CI-ELM). We study the enhanced and convex incremental ELM (ECI-ELM) algorithm, which is a combination of the last 2 versions. The main findings are that ECI-ELM is fast, accurate, and fully scalable when it operates in a parallel system of distributed memory workstations. Experimental simulations on several benchmark data sets demonstrate that the ECI-ELM is the most accurate among the existing I-ELM, EI-ELM, and CI-ELM algorithms. We also analyze the convergence as a function of the hidden neurons and demonstrate that ECI-ELM has the lowest error rate curve and converges much faster than the other algorithms in all of the data sets. The parallel simulations also reveal that the data parallel training of the ECI-ELM can guarantee simplicity and straightforward mappings and can deliver speedups and scale-ups very close to linear.

A Survey of Distributed and Parallel Extreme Learning Machine for Big Data

“…In addition to ensemble and matrix operations, there are other ways to implement distributed ELM. For example, iteration acceleration [82], using acceleration package on MATLAB [83], GPU acceleration [84], [85], using online sequential to realize distributed [86], and so on. Different from He [55], Xin [42].…”

“…Different from He [55], Xin [42]. Kokkinos [82] et al conducted the incremental version of ELM. Incremental ELM does not use direct matrix-matrix multiplicators, instead of adding neurons one by one, using each neuron to transmit one data, for direct parallelization.…”