A local linearized least squares algorithm for training feedforward neural networks

Stan, Ovidiu; Kamen, Edward W.

doi:10.1109/72.839017

Cited by 15 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Like localized EKF, a complex problem can be divided into multiple localized subproblems, each of which is solved by RLS. There exist some localized RLS algorithms, such as the local linearized LS (LLLS) method [301] and the block RLS (BRLS) algorithm [302].…”

Section: Recursive Least Squaresmentioning

confidence: 99%

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Leung

Mow

et al. 2022

Mathematics

View full text Add to dashboard Cite

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

show abstract

Section: Recursive Least Squaresmentioning

confidence: 99%

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Leung

Mow

et al. 2022

Mathematics

View full text Add to dashboard Cite

show abstract

“…In the "regression" submenu of SPSS, a very rich and powerful regression modelling function is provided. Based on the idea of least squares [31], for the regression model of more than one independent variable, the application of SPSS software can easily get the desired results. The weak impact on the working face is relatively small.…”

Section: Design Of Bp Neural Network Based On Matlabmentioning

confidence: 99%

A New Approach of Disaster Forecasting Based on Least Square Optimized Neural Network

et al. 2020

View full text Add to dashboard Cite

The evaluation of the risk is the prerequisite for the implementation of countermeasures in the prevention and control of rock burst, and the research on the fast forecast at scene of the rock burst is more important for the safety production of coal mine. Aiming at the problem that dynamic disasters caused by many factors and heterogeneity of coal and rock are difficult to predict in the process of coal mining, in this paper, the general law and the risk control factors of the rock burst are studied, a mathematical model based on the BP neural network was built according to the different actual mining conditions in the mining area, and the output layer has obtained the prediction result. Then, the results of the output samples after training were fitted by using SPSS software, and the fitting function was obtained by multiple least square fitting. Finally, the fitting results are checked by the data of actual coal mine dynamic disaster parameters. The prediction results show that the simulation results of BP neural network prediction model and the fitting function of the least square method can reduce the impact of subjective judgment on the prediction results, and the application of the fitting function can obtain the prediction results in the first time to ensure the construction safety. The method of on-site hazard assessment and inspection by using fitting function is simple and feasible and has high accuracy, which provides a new idea for the field prediction of rock burst.

show abstract

“…Thirdly, most of them are designed for training FNNs, and only a few are designed for training RNNs [31]. Lastly, most of them have high computational and space complexity, and some of them even require preserving one inverse autocorrelation matrix for each neuron [29], [33], [34]. As a result, the linear or recursive least squares methods seem to have been forgotten in DL.…”

Section: Introductionmentioning

confidence: 99%

Revisiting Recursive Least Squares for Training Deep Neural Networks

Zhang,

Song,

Zhou

et al. 2021

Preprint

View full text Add to dashboard Cite

Recursive least squares (RLS) algorithms were once widely used for training small-scale neural networks, due to their fast convergence. However, previous RLS algorithms are unsuitable for training deep neural networks (DNNs), since they have high computational complexity and too many preconditions. In this paper, to overcome these drawbacks, we propose three novel RLS optimization algorithms for training feedforward neural networks, convolutional neural networks and recurrent neural networks (including long short-term memory networks), by using the error backpropagation and our average-approximation RLS method, together with the equivalent gradients of the linear least squares loss function with respect to the linear outputs of hidden layers. Compared with previous RLS optimization algorithms, our algorithms are simple and elegant. They can be viewed as an improved stochastic gradient descent (SGD) algorithm, which uses the inverse autocorrelation matrix of each layer as the adaptive learning rate. Their time and space complexities are only several times those of SGD. They only require the loss function to be the mean squared error and the activation function of the output layer to be invertible. In fact, our algorithms can be also used in combination with other first-order optimization algorithms without requiring these two preconditions. In addition, we present two improved methods for our algorithms. Finally, we demonstrate their effectiveness compared to the Adam algorithm on MNIST, CIFAR-10 and IMDB datasets, and investigate the influences of their hyperparameters experimentally.

show abstract

A local linearized least squares algorithm for training feedforward neural networks

Cited by 15 publications

References 11 publications

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

A New Approach of Disaster Forecasting Based on Least Square Optimized Neural Network

Revisiting Recursive Least Squares for Training Deep Neural Networks

Contact Info

Product

Resources

About