A Novel Fast Feedforward Neural Networks Training Algorithm

Abstract

In this paper1 a new neural networks training algorithm is presented. The algorithm originates from the Recursive Least Squares (RLS) method commonly used in adaptive filtering. It uses the QR decomposition in conjunction with the Givens rotations for solving a normal equation - resulting from minimization of the loss function. An important parameter in neural networks is training time. Many commonly used algorithms require a big number of iterations in order to achieve a satisfactory outcome while other algorithms are effective only for small neural networks. The proposed solution is characterized by a very short convergence time compared to the well-known backpropagation method and its variants. The paper contains a complete mathematical derivation of the proposed algorithm. There are presented extensive simulation results using various benchmarks including function approximation, classification, encoder, and parity problems. Obtained results show the advantages of the featured algorithm which outperforms commonly used recent state-of-the-art neural networks training algorithms, including the Adam optimizer and the Nesterov’s accelerated gradient.

Detecting Anomalies in Advertising Web Traffic with the Use of the Variational Autoencoder

“…However, it requires different data, where the pro- portion between correct and abnormal data will be more even. The algorithm can also be implemented using other mechanisms of computational intelligence, for example other artificial neural network structures [22], using parallel computing mechanisms or statistical methods.…”

A Novel Approach to Type-Reduction and Design of Interval Type-2 Fuzzy Logic Systems

“…Such approximation of the maximum function, called a smooth maximum function, is used in this article to construct a novel type-reduction method for type-2 interval fuzzy logic. Properties of the smooth maximum function are useful for optimization of interval type-2 fuzzy systems with such techniques as gradient descent [2], conjugate gradient [3], or second-order gradient [1], smoothly optimizes the interval type-2 fuzzy logic system as new data is available. Consequently, in this paper, we derive a new adaptive structure of the interval type-2 fuzzy logic system equipped with smooth type-reduction.…”