Tensor train decompositions on recurrent networks

Abstract

Recurrent neural networks (RNN) such as long-short-term memory (LSTM) networks are essential in a multitude of daily live tasks such as speech, language, video, and multimodal learning. The shift from cloud to edge computation intensifies the need to contain the growth of RNN parameters. Current research on RNN shows that despite the performance obtained on convolutional neural networks (CNN), keeping a good performance in compressed RNNs is still a challenge. Most of the literature on compression focuses on CNNs using matrix product (MPO) operator tensor trains. However, matrix product state (MPS) tensor trains have more attractive features than MPOs, in terms of storage reduction and computing time at inference. We show that MPS tensor trains should be at the forefront of LSTM network compression through a theoretical analysis and practical experiments on NLP tasks.Since the beginning of the last decade, computational resources have taken a primary role in the advancing of artificial intelligence. With the advent of new architectures for neural networks, starting with AlexNet and convolutional neural networks, computational demand has accelerated to the point of doubling itself each hundred days [Amodei et al., 2019]. In fact from AlexNet to AlexZero, computing demand has increased 3000 times. This implies major challenges for resources such as memory. Neural network model size is a bottleneck. The use of memory and computer power for training neural nets is enormous. For example, AlexNet takes 1.5 weeks to train on ImageNet on one Nvidia TitanX GPU [Han et al., 2015]. Although this might seems like a long time, we have to consider that following the current trend in neural network modeling, a more sophisticated architecture, such as ResNet152, might take at least ten times longer to train. Model size is indeed a major challenge: larger model implies larger sizes, which implies higher energy consumption. In a recent tweet debate, Elliot Turner, the CEO and co-founder of Hologram AI, wrote that it costs $ Preprint. Under review.