An adaptive ensemble of on-line Extreme Learning Machines with variable forgetting factor for dynamic system prediction

Abstract

A demand for predictive models for on-line estimation of variables is increasing in industry. As industrial processes are timevarying, on-line learning algorithms should be adaptive to capture process changes. On-line ensemble methods have been shown to provide better generalization performance than single models in changing environments. However, most on-line ensembles do not include and exclude models during on-line operation. As a result, the ensembles have limited adaptation capability. Moreover, a higher performance can be obtained by combining a selected set of most relevant models of the ensemble for the current situation, rather than combining all the models. This paper proposes a new on-line learning ensemble of regressor models using an ordered aggregation (OA) technique which is able to provide on-line predictions of variables in changing environments. OA dynamically selects an optimal size and composition of a subset of models based on the minimization of the ensemble error on the newest sample. The proposed strategy overcomes the problem of defining the optimal ensemble size, and in most cases it obtains better performance than aggregating all the models. Models are added or removed for assuring adaptation of the ensemble in changing environments. Furthermore, this paper proposes and integrates a new on-line Extreme Learning Machine (ELM) neural network model with variable forgetting factor (FF) using the directional FF method which shows superior performance in industrial applications when compared to the well-known On-line Sequential ELM (OS-ELM) algorithm. Experiments are reported to demonstrate the performance and effectiveness of the proposed methods.

Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

“…Moreover, with the help of the adaptive forgetting mechanism, the AFGR-OSELM is capable of depicting the dynamic characteristics of the parameter-varying chaotic system more accurately; thus it achieves better prediction performances than the R-OSELM. In addition, we notice that the DFF-OSELM does not behave well in this simulation; the possible reason is that the default parameters recommended by [23] are not suitable for this task, and it may imply that the DFF-OSELM is problem specific.…”

“…The experimental results of the proposed algorithm are also compared with those of the OSELM [6], R-OSELM [16], FR-OSELM [24], and DFF-OSELM [23]. All of the five algorithms use the same sigmoidal additive activation function (a, , x) = 1/(1 + exp(−(a ⋅ x + ))), where the input weights a and the biases are randomly selected from the range [−1, 1].…”

“…Since the hidden nodes number is an important parameter which may greatly affect the learning accuracy and generalization performance of the neural network, for each algorithm we set = 20, = 50, and = 100, respectively, for experiments, and the corresponding numbers of training data for initialization are taken as 50, 100, and 200, respectively. Besides, other parameters for each algorithm are set as follows: for R-OSELM, = 10 −8 ; for FR-OSELM, = 10 −8 , = 0.98; for DFF-OSELM, all the parameters are set the same as in [23], and an extra regularization parameter, = 10 −8 , is added to stabilize the algorithm; for our AFGR-OSELM, = 10 −8 , = 0.1, + = 0.999, − = 0.8, the initial value for is set as 0.995, and the S and can be simply initialized to identity matrix and zero vector, respectively. Table 1 presents the mean and standard deviations (SD) of the prediction RMSE over 30 independent trials of the five models for performing different prediction steps on the simulated time-varying system.…”

“…Analogously, the concept of FF has also been introduced into the OSELM for dynamic system estimation and prediction [22][23][24]. In [22], a low complexity adaptive forgetting factor OSELM (LAFF-OSELM) algorithm was proposed for time-varying nonlinear system identification, and it was found that the proposed algorithm can deal with the variation effectively by adaptive forgetting mechanism.…”

“…In [22], a low complexity adaptive forgetting factor OSELM (LAFF-OSELM) algorithm was proposed for time-varying nonlinear system identification, and it was found that the proposed algorithm can deal with the variation effectively by adaptive forgetting mechanism. In [23], the authors presented a new variable forgetting factor OSELM using the directional FF method (DFF-OSELM), which achieved superior prediction performance in industrial applications when compared to the OSELM algorithm. However, in both LAFF-OSELM and DFF-OSELM, the intrinsic ill-posed problem is not considered, so they may encounter a potential instability.…”