A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Abstract

In extreme learning machines (ELM) the hidden-layer coefficients are randomly set and fixed, while the output-layer coefficients of the neural network are computed by a least squares method. The randomlyassigned coefficients in ELM are known to influence its performance and accuracy significantly. In this paper we present a modified batch intrinsic plasticity (modBIP) method for pre-training the random coefficients in the ELM neural networks. The current method is devised based on the same principle as the batch intrinsic plasticity (BIP) method, namely, by enhancing the information transmission in every node of the neural network. It differs from BIP in two prominent aspects. First, modBIP does not involve the activation function in its algorithm, and it can be applied with any activation function in the neural network. In contrast, BIP employs the inverse of the activation function in its construction, and requires the activation function to be invertible (or monotonic). The modBIP method can work with the oftenused non-monotonic activation functions (e.g. Gaussian, swish, Gaussian error linear unit, and radial-basis type functions), with which BIP breaks down. Second, modBIP generates target samples on random intervals with a minimum size, which leads to highly accurate computation results when combined with ELM. The combined ELM/modBIP method is markedly more accurate than ELM/BIP in numerical simulations. Ample numerical experiments are presented with shallow and deep neural networks for function approximation and boundary/initial value problems with partial differential equations. They demonstrate that the combined ELM/modBIP method produces highly accurate simulation results, and that its accuracy is insensitive to the random-coefficient initializations in the neural network. This is in sharp contrast with the ELM results without pre-training of the random coefficients.

Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks

“…Therefore, in order to compute the Jacobian matrix we first solve equation ( 4) for β LS by the linear least squares method, and then use (10) to compute J 0 (θ). We then compute J 1 (θ) by equations ( 13) and (14). Finally the Jacobian matrix ∂r ∂θ is computed by equation ( 12).…”

“…As in our previous works [13,14,17], we employ a fixed seed value for the random number generators in the numerical experiments in each subsection, so that the reported results here can be exactly reproducible. We use the same seed for the random number generators from the Tensorflow library and from the numpy package.…”

“…Their weakness lies in the limited accuracy and the high computational cost (long network-training time). Another promising class of neural network-based methods for computational PDEs has recently appeared [13,14,18,21,17], which are based on a type of randomized neural networks called extreme learning machines (ELM) [29,30]. With these methods the weight/bias coefficients in the hidden layers of the neural network are set to random values and are fixed.…”

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

“…The ELM type idea has also been developed for nonlinear problems; see e.g. [10,11] for solving stationary and time-dependent nonlinear PDEs in which the neural network is trained by a nonlinear least squares method. Following [11], we broadly refer to the artificial neural network-based methods exploiting these strategies as ELM type methods, including those employing neural networks with multiple hidden layers and those for nonlinear problems [49,51,10,11,17].…”

“…The pursuit for higher accuracy and more competitive performance with neural networks for computational PDEs has led us in [10,11] to explore randomized neural networks (including ELM) [45,18]. Since optimizing the entire set of weight/bias coefficients in the neural network can be extremely hard and costly, perhaps randomly assigning and fixing a subset of the network's weights will make the resultant optimization task of network training simpler, and ideally linear, without severely sacrificing the achievable approximation capacity.…”