Achieving Online Regression Performance of LSTMs With Simple RNNs

Abstract

Recurrent neural networks (RNNs) are widely used for online regression due to their ability to generalize nonlinear temporal dependencies. As an RNN model, long short-term memory networks (LSTMs) are commonly preferred in practice, as these networks are capable of learning long-term dependencies while avoiding the vanishing gradient problem. However, due to their large number of parameters, training LSTMs requires considerably longer training time compared to simple RNNs (SRNNs). In this article, we achieve the online regression performance of LSTMs with SRNNs efficiently. To this end, we introduce a first-order training algorithm with a linear time complexity in the number of parameters. We show that when SRNNs are trained with our algorithm, they provide very similar regression performance with the LSTMs in two to three times shorter training time. We provide strong theoretical analysis to support our experimental results by providing regret bounds on the convergence rate of our algorithm. Through an extensive set of experiments, we verify our theoretical work and demonstrate significant performance improvements of our algorithm with respect to LSTMs and the other state-of-the-art learning models.

A mathematical framework for improved weight initialization of neural networks using Lagrange multipliers

“…Moreover, we assume that the last layer L of this neural network is a fully connected layer. This is often the case for neural networks that solve a regression problem, e.g., Kim et al (2020), Vural, Ilhan, Yilmaz, Ergüt, andKozat (2021) and Yan (2012). A schematic overview of this last layer is in Fig.…”

“…We assume that the considered neural network applies a linear activation function to the output, i.e., the output y L i of the last layer directly is the estimated label ŷi of training sample i. This linear activation function is also often applied in neural networks that solve a regression problem, e.g., Kim et al (2020), Vural et al (2021) and Yan (2012). The vector with estimated labels for the…”