Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

Abstract

While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10-50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50-400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30-50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.

Discriminative segmental cascades for feature-rich phone recognition

“…test ER (%) Gaussian mixture model (GMM) [23] 26.3 SVM [23] 22.4 Hierarchical GMM [22] 21.0 Discriminative hierarchical GMM [24] 16.8 SVM with deep scattering spectrum [25] 15.9 our CNN ensemble 15.0 the softmax [21]. Each frame is labeled with its segment label and one additional label from a neighboring segment.…”

Mean-normalized stochastic gradient for large-scale deep learning

“…On the other hand, all layers can be compressed using Xue et al's approach, which accelerates recognition, but makes training even more expensive. The value of the linear bottleneck structure as a regularization method has not been identified in [15,16]. Our experiments show that using MN-SGD allows for a factorization of all layers when training from scratch.…”

“…Recently, a more sophisticated approach for reducing the size of DNNs has been proposed. Sainath et al [15] factored the weight matrices into the product of two smaller matrices. This is equivalent with inserting a linear bottleneck between two layers of the network.…”

“…In this work, we investigate a recently proposed model structure [15,16] that allows for reducing model size and thus improves generalization. Furthermore, these smaller models reduce computational costs in both training and decoding.…”

Metric learning with rank and sparsity constraints

“…A sparse A has computational benefits like low storage and computational complexity. Consequently, this work could be useful in sparse lowrank matrix factorization which has numerous applications in machine learning including learning [7] and deep neural networks (deep learning) [8] and autoencoding. This work is also related to optimizing projection matrices introduced in [9].…”