Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Huynh,; Won, Yonggwan

doi:10.1016/j.patrec.2011.07.016

Cited by 103 publications

(52 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once the singular or ill-posed problem occurs, the generalization performance of OSELM will deteriorate significantly. In order to overcome this problem, Huynh and Won [16] proposed a regularized OSELM (R-OSELM) by using the Tikhonov regularization. The learning procedure of the R-OSELM is almost the same as the OSELM, just adding a regularization item to the autocorrelation matrix H 푇 H to avoid the singular or ill-posed problem so as to improve the stability of the algorithm.…”

Section: R-oselmmentioning

confidence: 99%

“…Despite an excellent online learning algorithm, the OSELM may still suffer from a drawback of instability due to the potential ill-conditioned matrix inversion, and its stability and generalization performance could be greatly influenced once the autocorrelation matrix of the hidden layer output matrix is singular or ill-conditioning. Regularization technique is an effective way to cope with the ill-posed problem, and a regularized OSELM (R-OSELM) based on the biobjective optimization with Tikhonov regularization was proposed in [16]. The R-OSELM successfully overcomes the potential ill-posed problem and tends to provide good generalization performance and stability, and it has become a practical online modeling method in real applications.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

Wei

Tang

et al. 2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Many real world applications are of time-varying nature and an online learning algorithm is preferred in tracking the realtime changes of the time-varying system. Online sequential extreme learning machine (OSELM) is an excellent online learning algorithm, and some improved OSELM algorithms incorporating forgetting mechanism have been developed to model and predict the time-varying system. But the existing algorithms suffer from a potential risk of instability due to the intrinsic ill-posed problem; besides, the adaptive tracking ability of these algorithms for complex time-varying system is still very weak. In order to overcome the above two problems, this paper proposes a novel OSELM algorithm with generalized regularization and adaptive forgetting factor (AFGR-OSELM). In the AFGR-OSELM, a new generalized regularization approach is employed to replace the traditional exponential forgetting regularization to make the algorithm have a constant regularization effect; consequently the potential illposed problem of the algorithm can be completely avoided and a persistent stability can be guaranteed. Moreover, the AFGR-OSELM adopts an adaptive scheme to adjust the forgetting factor dynamically and automatically in the online learning process so as to better track the dynamic changes of the time-varying system and reduce the adverse effects of the outdated data in time; thus it tends to provide desirable prediction results in time-varying environment. Detailed performance comparisons of AFGR-OSELM with other representative algorithms are carried out using artificial and real world data sets. The experimental results show that the proposed AFGR-OSELM has higher prediction accuracy with better stability than its counterparts for predicting time-varying system.

show abstract

Section: R-oselmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

Wei

Tang

et al. 2018

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…It has been reported that ELM might run into an ill-posed problem when it is over parameterized or when not properly initialized [11], [12], [13], [14]. A few attempts have been made to improve the regularization behavior or ELM [13] and OS-ELM [14].…”

Section: Stability Advantage Of L-elmmentioning

confidence: 99%

“…A few attempts have been made to improve the regularization behavior or ELM [13] and OS-ELM [14]. However, when the data is being processed 1-by-1 (as in the case of system identification), the regularization improvement suggested by [14] was not found to improve the situation. Such an unstable parametric evolution can cause fatal problems when such online models are used in decision making.…”

Section: Stability Advantage Of L-elmmentioning

confidence: 99%

“…In spite of its known advantages (see section II), an over-parameterized ELM suffers from the ill-conditioning problem when recursive least squares type update is performed (as in OS-ELM). This results in poor regularization behavior [11], [12], [13], [14], and more importantly, leads to an unbounded growth of the model parameters and predictions. The goal of this paper is to address the above issue by developing a stable online learning algorithm for ELM using Lyapunov approach which guarantees the boundedness of the model parameters as well as predictions even when the model is over-parameterized.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A lyapunov based stable online learning algorithm for nonlinear dynamical systems using extreme learning machines

Janakiraman

2013

The 2013 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Abstract-Extreme Learning Machine (ELM) is a promising learning scheme for nonlinear classification and regression problems and has shown its effectiveness in the machine learning literature. ELM represents a class of generalized single hidden layer feed-forward networks (SLFNs) whose hidden layer parameters are assigned randomly resulting in an extremely fast learning speed along with superior generalization performance. It is well known that the online sequential learning algorithm (OS-ELM) based on recursive least squares [1] might result in ill-conditioning of the Hessian matrix and hence instability in the parameter estimation. To address this issue, the stability theory of Lyapunov is utilized to develop an online learning algorithm for temporal data from dynamic systems and time series. The developed algorithm results in parameter estimation that is globally asymptotically stable. Simulations results of the developed algorithm compared against online sequential ELM (OS-ELM) and the offline batch learning ELM (O-ELM) show that the online Lyapunov ELM algorithm can perform online learning at reduced computation, comparable accuracy and with a guarantee on the boundedness of the estimated system.

show abstract

GPU‐accelerated and mixed norm regularized online extreme learning machine

Polat

Kayhan

2022

Concurrency and Computation

View full text Add to dashboard Cite

Extreme learning machine (ELM) is a prominent example of neural network with its fast training speed, and good prediction performance. An online version of ELM called online sequential extreme learning machine (OS-ELM) has also been proposed for the sequential training. Combined with the need for regularization to prevent over-fitting in addition to the large number of neurons required in the hidden layer, OS-ELM demands huge amount of computation power for the large-scale data. In this article, a mixed norm (l 2,1 ) regularized online machine learning algorithm (MRO-ELM) that is based on alternating direction method of multipliers (ADMM) is proposed. A linear combination of the mixed norm and the Frobenius norm regularization is applied using the ADMM framework and update formulas are derived. Graphics processing unit (GPU) accelerated version of MRO-ELM (GPU-MRO-ELM) is also proposed to reduce the training time by processing appropriate parts in parallel using the implemented custom kernels. In addition, a novel automatic hyper-parameter tuning method is incorporated to GPU-MRO-ELM using progressive validation with GPU acceleration.The experimental results show that the MRO-ELM algorithm and its GPU version outperform OS-ELM in terms of training speed, and testing accuracy. Also, compared to the cross validation, the proposed automatic hyper-parameter tuning demonstrates dramatical reduction in the tuning time.

show abstract

Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Cited by 103 publications

References 15 publications

Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

A lyapunov based stable online learning algorithm for nonlinear dynamical systems using extreme learning machines

GPU‐accelerated and mixed norm regularized online extreme learning machine

Contact Info

Product

Resources

About