Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Ye, Jinmian; Wang, Linnan; Li, Guangxi; Chen, Di; Zhe, Shandian; Chu, Xinqi; Xu, Zenglin

doi:10.1109/cvpr.2018.00977

Cited by 110 publications

(79 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the empirical results, TT-format are able to reduce the number of parameters significantly and retain the model performance at the same time. Recent work from [36] used block decompositions to represent the RNN weight matrices.…”

Section: Resultsmentioning

confidence: 99%

Tensor Decomposition for Compressing Recurrent Neural Network

Tjandra

Sakti

Nakamura

2018

2018 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

In the machine learning fields, Recurrent Neural Network (RNN) has become a popular architecture for sequential data modeling. However, behind the impressive performance, RNNs require a large number of parameters for both training and inference. In this paper, we are trying to reduce the number of parameters and maintain the expressive power from RNN simultaneously. We utilize several tensor decompositions method including CANDECOMP/PARAFAC (CP), Tucker decomposition and Tensor Train (TT) to re-parameterize the Gated Recurrent Unit (GRU) RNN. We evaluate all tensor-based RNNs performance on sequence modeling tasks with a various number of parameters. Based on our experiment results, TT-GRU achieved the best results in a various number of parameters compared to other decomposition methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Tensor Decomposition for Compressing Recurrent Neural Network

Tjandra

Sakti

Nakamura

2018

2018 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…Our study concentrated on the ability to estimate dose in heterogeneous geometries, and no effort was made in improving the model efficiency. Various model compression techniques, for example, pruning, quantization, and tensor decomposition methods (achieving low-rank structures in the weight matrices), [51][52][53] may substantially lower the number of parameters in fully connected layers. 54,55 The efficiency of the model can be further enhanced through fine-tuning of the model architecture.…”

Section: In This Paper We Have Demonstrated the General Feasibility mentioning

confidence: 99%

Long short‐term memory networks for proton dose calculation in highly heterogeneous tissues

et al. 2021

View full text Add to dashboard Cite

To investigate the feasibility and accuracy of proton dose calculations with artificial neural networks (ANNs) in challenging three-dimensional (3D) anatomies. Methods: A novel proton dose calculation approach was designed based on the application of a long short-term memory (LSTM) network. It processes the 3D geometry as a sequence of two-dimensional (2D) computed tomography slices and outputs a corresponding sequence of 2D slices that forms the 3D dose distribution. The general accuracy of the approach is investigated in comparison to Monte Carlo reference simulations and pencil beam dose calculations. We consider both artificial phantom geometries and clinically realistic lung cases for three different pencil beam energies. Results: For artificial phantom cases, the trained LSTM model achieved a 98.57% γ-index pass rate ([1%, 3 mm]) in comparison to MC simulations for a pencil beam with initial energy 104.25 MeV. For a lung patient case, we observe pass rates of 98.56%, 97.74%, and 94.51% for an initial energy of 67.85, 104.25, and 134.68 MeV, respectively. Applying the LSTM dose calculation on patient cases that were fully excluded from the training process yields an average γ-index pass rate of 97.85%. Conclusions: LSTM networks are well suited for proton dose calculation tasks. Further research, especially regarding model generalization and computational performance in comparison to established dose calculation methods, is warranted.

show abstract

“…-Low rank factorization: [10,36] -Factorized embedding parameterization: [19] -Block-Term tensor decomposition: [23,38] -Singular Value Decomposition: [37] -Joint factorization of recurrent and inter-layer weight matrices: [28] -Tensor train decomposition: [10,17] -Sparse factorization: [6] • [11] • Applications: In this section, we will discuss application and success of various model compression methods across various popular NLP tasks like Language modeling, Machine translation, Summarization, Sentiment analysis, Question answering, Natural language inference, Paraphrasing, Image captioning, Handwritten character recognition. • Summary and future trends.…”

Section: Tutorial Outlinementioning

confidence: 99%

Compression of Deep Learning Models for NLP

Gupta

Varma

Damani

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

In recent years, the fields of NLP and information retrieval have made tremendous progress thanks to deep learning models like RNNs and LSTMs, and Transformer [35] based models like BERT [9]. But these models are humongous in size. Real world applications however demand small model size, low response times and low computational power wattage. We will discuss six different types of methods (pruning, quantization, knowledge distillation, parameter sharing, matrix decomposition, and other Transformer based methods) for compression of such models to enable their deployment in real industry NLP projects. Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this tutorial is very timely. We will organize related work done by the 'deep learning for NLP' community in the past few years and present it as a coherent story. CCS CONCEPTS • Computing methodologies → Neural networks; Machine learning; Natural language processing; • Theory of computation → Models of learning.

show abstract

Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

Cited by 110 publications

References 43 publications

Tensor Decomposition for Compressing Recurrent Neural Network

Tensor Decomposition for Compressing Recurrent Neural Network

Long short‐term memory networks for proton dose calculation in highly heterogeneous tissues

Compression of Deep Learning Models for NLP

Contact Info

Product

Resources

About