Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers

Bemporad, Alberto

doi:10.1016/j.automatica.2023.111183

Cited by 3 publications

(1 citation statement)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second-order methods, however, can be used to mitigate these drawbacks by utilizing, in their learning procedures, the information contained in the curvature of the loss surface, which provably accelerates convergence. The second-order methods for learning RNNs have historically been used in two ways: 1) the use of second-order optimization algorithms, such as generalized Gauss-Newton (GGN), Levenberg-Marquardt (LM), and conjugate gradient (CG) algorithms (see [12], [13], [14], [15], [16], [17], [18], [19]) and 2) using nonlinear sequential stateestimation techniques, such as extended Kalman filter (EKF) method (see [20], [21], [22], [23], [24], [25]). In the first approach, the second-order information is captured through Hessian (or approximate Hessian) computations, while in the second approach, the second-order information is computed recursively as a prediction-error covariance matrix.…”

Section: Introductionmentioning

confidence: 99%

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks

Adeoye,

Bemporad

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

This article considers the two-stage approach to solving a partially observable Markov decision process (POMDP): the identification stage and the (optimal) control stage. We present an inexact sequential quadratic programming framework for recurrent neural network learning (iSQPRL) for solving the identification stage of the POMDP, in which the true system is approximated by a recurrent neural network (RNN) with dynamically consistent overshooting (DCRNN). We formulate the learning problem as a constrained optimization problem and study the quadratic programming (QP) subproblem with a convergence analysis under a restarted Krylov-subspace iterative scheme that implicitly exploits the structure of the associated Karush-Kuhn-Tucker (KKT) subsystem. In the control stage, where a feedforward neural network (FNN) controller is designed on top of the RNN model, we adapt a generalized Gauss-Newton (GGN) algorithm that exploits useful approximations to the curvature terms of the training data and selects its mini-batch step size using a known property of some regularization function. Simulation results are provided to demonstrate the effectiveness of our approach.

show abstract

Section: Introductionmentioning

confidence: 99%

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks

Adeoye,

Bemporad

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

Nonlinear sparse variational Bayesian learning based model predictive control with application to PEMFC temperature control

Zhang,

Wang,

et al. 2024

Control Engineering Practice

View full text Add to dashboard Cite

An Enhanced Tunicate Swarm Algorithm with Symmetric Cooperative Swarms for Training Feedforward Neural Networks

Du,

Zhang

2024

Symmetry

View full text Add to dashboard Cite

The input layer, hidden layer, and output layer are three models of neural processors that comprise feedforward neural networks. In this paper, an enhanced tunicate swarm algorithm based on a differential sequencing alteration operator (ETSA) with symmetric cooperative swarms is presented to train feedforward neural networks. The objective is to accomplish minimum classification errors and the most appropriate neural network layout by regulating the layers’ connection weights and neurons’ deviation thresholds according to the transmission error between the anticipated input and the authentic output. The TSA mimics jet motorization and swarm scavenging to mitigate directional collisions and to maintain the greatest solution that is customized and regional. However, the TSA exhibits the disadvantages of low computational accuracy, a slow convergence speed, and easy search stagnation. The differential sequencing alteration operator has adaptable localized extraction and search screening to broaden the identification scope, enrich population creativity, expedite computation productivity, and avoid search stagnation. The ETSA integrates exploration and exploitation to mitigate search stagnation, which has sufficient stability and flexibility to acquire the finest solution. The ETSA was distinguished from the ETTAO, EPSA, SABO, SAO, EWWPA, YDSE, and TSA by monitoring seventeen alternative datasets. The experimental results confirm that the ETSA maintains profound sustainability and durability to avoid exaggerated convergence, locate the acceptable transmission error, and equalize extraction and prospection to yield a faster convergence speed, superior calculation accuracy, and greater categorization accuracy.

show abstract

Training recurrent neural networks by sequential least squares and the alternating direction method of multipliers

Cited by 3 publications

References 28 publications

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks

Nonlinear sparse variational Bayesian learning based model predictive control with application to PEMFC temperature control

An Enhanced Tunicate Swarm Algorithm with Symmetric Cooperative Swarms for Training Feedforward Neural Networks

Contact Info

Product

Resources

About