On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Abstract

We consider the use of extreme learning machines (ELM) for computational partial differential equations (PDE). In ELM the hidden-layer coefficients in the neural network are assigned to random values generated on [−Rm, Rm] and fixed, where Rm is a user-provided constant, and the output-layer coefficients are trained by a linear or nonlinear least squares computation. We present a method for computing the optimal or near-optimal value of Rm based on the differential evolution algorithm. This method seeks the optimal Rm by minimizing the residual norm of the linear or nonlinear algebraic system that results from the ELM representation of the PDE solution and that corresponds to the ELM least squares solution to the system. It amounts to a pre-processing procedure that determines a near-optimal Rm, which can be used in ELM for solving linear or nonlinear PDEs. The presented method enables us to illuminate the characteristics of the optimal Rm for two types of ELM configurations: (i) Single-Rm-ELM, corresponding to the conventional ELM method in which a single Rm is used for generating the random coefficients in all the hidden layers, and (ii) Multi-Rm-ELM, corresponding to a modified ELM method in which multiple Rm constants are involved with each used for generating the random coefficients of a different hidden layer. We adopt the optimal Rm from this method and also incorporate other improvements into the ELM implementation. In particular, here we compute all the differential operators involving the output fields of the last hidden layer by a forward-mode auto-differentiation, as opposed to the reverse-mode auto-differentiation in a previous work. These improvements significantly reduce the network training time and enhance the ELM performance. We systematically compare the computational performance of the current improved ELM with that of the finite element method (FEM), both the classical second-order FEM and the high-order FEM with Lagrange elements of higher degrees, for solving a number of linear and nonlinear PDEs. It is shown that the current improved ELM far outperforms the classical FEM. Its computational performance is comparable to that of the high-order FEM for smaller problem sizes, and for larger problem sizes the ELM markedly outperforms the high-order FEM.

Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks

“…We observe that, for smooth field solutions, the VarPro errors decrease exponentially as the number of collocation points or the number of output-layer coefficients in the neural network increases, which is reminiscent of the spectral convergence of traditional high-order methods [32,60,69,16,12,11,38,66,67]. We also compare extensively the performance of the current VarPro method with that of the ELM method from [13,17]. The numerical results show that, under identical conditions and network configurations, the VarPro method is considerably more accurate than the ELM method, especially when the size of the neural network is small.…”

“…Remark 2.6. It would be interesting to compare the current VarPro method with the extreme learning machine (ELM) method from [13,17] for solving PDEs. With ELM, the weight/bias coefficients in all the hidden layers of the neural network are pre-set to random values and are fixed, while the output-layer coefficients are computed by the linear least squares method for solving linear PDEs and by the nonlinear least squares method for solving nonlinear PDEs [13].…”

“…Their weakness lies in the limited accuracy and the high computational cost (long network-training time). Another promising class of neural network-based methods for computational PDEs has recently appeared [13,14,18,21,17], which are based on a type of randomized neural networks called extreme learning machines (ELM) [29,30]. With these methods the weight/bias coefficients in the hidden layers of the neural network are set to random values and are fixed.…”

“…In addition, the computational performance of the ELM-type method from [13] is observed to be comparable to or exceed that of the classical finite element method (FEM). Further extensions and improvements to the ELM method of [13] have been documented in [17] recently, which compares systematically the improved method with the classical and high-order FEM for solving a number of linear and nonlinear PDEs. The improved ELM method outperforms the classical second-order FEM by a considerable margin, and it outcompetes the high-order FEM when the problem size is not very small [17].…”

“…In Section 3 we present several numerical examples with linear and nonlinear PDEs to demonstrate the accuracy of the VarPro method developed herein. The performance of the current VarPro method is compared extensively with that of the ELM method from [13,17]. Section 4 then concludes the presentation with some closing remarks and comments on the presented method.…”