Efficient Language Modeling with Automatic Relevance Determination in Recurrent Neural Networks

Abstract

Reduction of the number of parameters is one of the most important goals in Deep Learning. In this article we propose an adaptation of Doubly Stochastic Variational Inference for Automatic Relevance Determination (DSVI-ARD) for neural networks compression. We find this method to be especially useful in language modeling tasks, where large number of parameters in the input and output layers is often excessive. We also show that DSVI-ARD can be applied together with encoder-decoder weight tying allowing to achieve even better sparsity and performance. Our experiments demonstrate that more than 90% of the weights in both encoder and decoder layers can be removed with a minimal quality loss. * These two authors contributed equally; the ordering of their names was chosen arbitrarily. The work was done when the first author was an intern at the Samsung R&D Institute.

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

“…Earlier works have focused on inducing sparsity in standard feed-forward neural networks. Yet, Bayesian pruning methods have also been successfully applied to recurrent neural networks (RNNs) [Kodryan et al 2019;Lobacheva et al 2018]. Lobacheva et al [2018] use Sparse VD to prune individual weights of an LSTM or follow the approach from Louizos et al [2017] to sparsify neurons or gates and show results on text classification or language modeling problems.…”

“…Lobacheva et al [2018] use Sparse VD to prune individual weights of an LSTM or follow the approach from Louizos et al [2017] to sparsify neurons or gates and show results on text classification or language modeling problems. Kodryan et al [2019] use instead the Automatic Relevance Determination (ARD) framework, in which a zero-mean element-wise factorized Gaussian prior distribution over the parameters is used, together with a corresponding Gaussian factorized posterior, such that a closed-form expression of the KL divergence term of the variational lower bound is obtained. Subsequently, the Doubly Stochastic Variational Inference (DSVI) method is used to maximize the variational lower bound and the weights for which the prior variances are lower than a certain threshold are set to zero.…”