On robust randomized neural networks for regression: a comprehensive review and evaluation

Freire, Ananda L.; Neto, Ajalmar R. da Rocha; Barreto, Guilherme A.

doi:10.1007/s00521-020-04994-5

Cited by 16 publications

(11 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The random coefficients in the hidden layers of the neural network are crucial to the performance and accuracy of ELM [30,29,13,6]. It has been observed from the numerical experiments in [6] that the ELM accuracy can be influenced strongly by the maximum magnitude of the random coefficients (i.e.…”

Section: Extreme Learning Machine and Random Coefficientsmentioning

confidence: 99%

“…[6]). ELM is one type of random-weight neural networks [37,13], which randomly assign and fix a subset of the network's weights so that the resultant optimization task of training the neural network can be simpler, and often linear, for example, formulated as a linear least squares problem. Randomization can be applied to both feed-forward and recurrent networks, leading to methodologies such as the random vector functional link (RVFL) networks [32,21], the extreme learning machine [20,19], the no-propagation network [45], the echo-state network [22,27], and the liquid state machine [28].…”

Section: Introductionmentioning

confidence: 99%

“…The effects of the random coefficients on the performance of ELM and other random-weight neural networks have also been recognized in regression and classification problems other than scientific computing (see e.g. [30,29,49,41,9,13], among others).…”

Section: Introductionmentioning

confidence: 99%

“…In addition to the above works, other researchers have aimed to utilize the relationship between the input-data rank and the performance of randomized neural networks, or to pick the weights/bias based on the input data range and the activation function type, or to consider the numerical stabilities (see e.g. [1,9,13], among others).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Dong,

2021

Preprint

View full text Add to dashboard Cite

In extreme learning machines (ELM) the hidden-layer coefficients are randomly set and fixed, while the output-layer coefficients of the neural network are computed by a least squares method. The randomlyassigned coefficients in ELM are known to influence its performance and accuracy significantly. In this paper we present a modified batch intrinsic plasticity (modBIP) method for pre-training the random coefficients in the ELM neural networks. The current method is devised based on the same principle as the batch intrinsic plasticity (BIP) method, namely, by enhancing the information transmission in every node of the neural network. It differs from BIP in two prominent aspects. First, modBIP does not involve the activation function in its algorithm, and it can be applied with any activation function in the neural network. In contrast, BIP employs the inverse of the activation function in its construction, and requires the activation function to be invertible (or monotonic). The modBIP method can work with the oftenused non-monotonic activation functions (e.g. Gaussian, swish, Gaussian error linear unit, and radial-basis type functions), with which BIP breaks down. Second, modBIP generates target samples on random intervals with a minimum size, which leads to highly accurate computation results when combined with ELM. The combined ELM/modBIP method is markedly more accurate than ELM/BIP in numerical simulations. Ample numerical experiments are presented with shallow and deep neural networks for function approximation and boundary/initial value problems with partial differential equations. They demonstrate that the combined ELM/modBIP method produces highly accurate simulation results, and that its accuracy is insensitive to the random-coefficient initializations in the neural network. This is in sharp contrast with the ELM results without pre-training of the random coefficients.

show abstract

Section: Extreme Learning Machine and Random Coefficientsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Dong,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The pursuit for higher accuracy and more competitive performance with neural networks for computational PDEs has led us in [10,11] to explore randomized neural networks (including ELM) [45,18]. Since optimizing the entire set of weight/bias coefficients in the neural network can be extremely hard and costly, perhaps randomly assigning and fixing a subset of the network's weights will make the resultant optimization task of network training simpler, and ideally linear, without severely sacrificing the achievable approximation capacity.…”

mentioning

confidence: 99%

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Dong,

Yang

2021

Preprint

View full text Add to dashboard Cite

We consider the use of extreme learning machines (ELM) for computational partial differential equations (PDE). In ELM the hidden-layer coefficients in the neural network are assigned to random values generated on [−Rm, Rm] and fixed, where Rm is a user-provided constant, and the output-layer coefficients are trained by a linear or nonlinear least squares computation. We present a method for computing the optimal or near-optimal value of Rm based on the differential evolution algorithm. This method seeks the optimal Rm by minimizing the residual norm of the linear or nonlinear algebraic system that results from the ELM representation of the PDE solution and that corresponds to the ELM least squares solution to the system. It amounts to a pre-processing procedure that determines a near-optimal Rm, which can be used in ELM for solving linear or nonlinear PDEs. The presented method enables us to illuminate the characteristics of the optimal Rm for two types of ELM configurations: (i) Single-Rm-ELM, corresponding to the conventional ELM method in which a single Rm is used for generating the random coefficients in all the hidden layers, and (ii) Multi-Rm-ELM, corresponding to a modified ELM method in which multiple Rm constants are involved with each used for generating the random coefficients of a different hidden layer. We adopt the optimal Rm from this method and also incorporate other improvements into the ELM implementation. In particular, here we compute all the differential operators involving the output fields of the last hidden layer by a forward-mode auto-differentiation, as opposed to the reverse-mode auto-differentiation in a previous work. These improvements significantly reduce the network training time and enhance the ELM performance. We systematically compare the computational performance of the current improved ELM with that of the finite element method (FEM), both the classical second-order FEM and the high-order FEM with Lagrange elements of higher degrees, for solving a number of linear and nonlinear PDEs. It is shown that the current improved ELM far outperforms the classical FEM. Its computational performance is comparable to that of the high-order FEM for smaller problem sizes, and for larger problem sizes the ELM markedly outperforms the high-order FEM.

show abstract

Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine

Dong

2023

J Sci Comput

View full text Add to dashboard Cite

On robust randomized neural networks for regression: a comprehensive review and evaluation

Cited by 16 publications

References 55 publications

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine

Contact Info

Product

Resources

About