Importance Estimation for Neural Network Pruning

Molchanov, Pavlo; Mallya, Arun; Tyree, Stephen; Frosio, Iuri; Kautz, Jan

doi:10.1109/cvpr.2019.01152

Cited by 665 publications

(522 citation statements)

References 22 publications

Supporting

Mentioning

430

Contrasting

Unclassified

Order By: Relevance

“…In this section, we evaluate the proposed pruning methodology in terms of accuracy, compression achieved, as well as time and computational complexity of the pruning process. We compare our technique with several existing and state-ofthe-art pruning methods such as [5], [9], [16], [17], [32], [33]. We have used three different datasets namely CIFAR-10, CIFAR-100 and ImageNet and trained them on the VGGNet and ResNet architectures.…”

Section: Resultsmentioning

confidence: 99%

“…Most of the existing pruning techniques usually involve the following three stages [4]: 1) Training an over-parameterized network till convergence, 2) Pruning based on predefined criteria, and 3) Fine-tuning/re-training to regain accuracy. The main limitation with this three-stage process adopted by existing pruning methods is that the pruning and fine-tuning stages, iterative in most cases [5]- [9], [38], impose considerable additional computation (hence, time and energy) requirements on top of the compute-heavy training stage. Based on the structure and criteria used for pruning, most of the previous works on network pruning also suffer from one or more of the following problems: ples of unstructured pruning methods include [5]- [11].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks

et al. 2020

View full text Add to dashboard Cite

The enormous inference cost of deep neural networks can be mitigated by network compression. Pruning connections is one of the predominant approaches used for network compression. However, existing pruning techniques suffer from one or more of the following limitations: 1) They increase the time and energy consumed by the compute-heavy training stage due to the addition of the pruning and fine-tuning steps, 2) They prune layer-wise based on local information about a particular layer's statistics, ignoring the effect of error propagation through the network, 3) They lack an efficient means to determine the global importance of channels, 4) Due to the use of unstructured pruning, they may not lead to any energy advantage when implemented on mainstream platforms (GPUs and TPUs), requiring specialized hardware to reap the benefits. To address the above issues, we present a simple-yet-effective methodology for gradual channel pruning while training using a data-driven metric referred to as feature relevance score. The proposed technique eliminates the need for additional retraining by pruning the least important channels in a structured manner at fixed intervals during the regular training phase. Pruning is guided by feature relevance scores, which help in efficiently evaluating the contribution of each channel towards the discriminative power of the network. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet using datasets such as CIFAR-10, CIFAR-100, and ImageNet, and successfully achieve significant model compression while trading off less than 1% accuracy INDEX TERMS Convolutional Neural Networks (CNNs), Deep learning, Efficient deep learning, Neural Networks, Model architecture, Model compression, Network design, Relevance scores, Structured pruning.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Fortunately, some studies have focused on the complexity reduction of STFT [25], which could simplify the computational process significantly. Besides, with the rapid developing of deep learning, the complexity of the network in TFDNet would be further reduced by the optimization methods such as pruning [26]. Most importantly, the construction of TFDNet may provide some inspiration about exploiting more image processing and analysis methods which are suitable to tackle problems in UVLC system.…”

Section: Results and Analysismentioning

confidence: 99%

Nonlinear Resilient Learning Method Based on Joint Time-Frequency Image Analysis in Underwater Visible Light Communication

Chen

Zhao

et al. 2020

IEEE Photonics J.

View full text Add to dashboard Cite

In this paper, we propose a novel nonlinear resilient learning post equalizer named TFDNet in UVLC system. Unlike the traditional deep neural network (DNN) based post equalizers which merely consider the time domain, the proposed TFDNet exploits timefrequency image analysis which considers the time and frequency domains simultaneously and transforms the signal into 2D time-frequency image, which is further learned by neural network. Experimental results demonstrate that TFDNet outperforms Volterra and DNN based methods for compensating nonlinear distortions through a 1.2 m underwater channel using 64 quadrature amplitude modulation-carrierless amplitude modulation (64QAM-CAP). Even under severe nonlinear distortions where Volterra and DNN cannot work, TFDNet retains valid bit error rate (BER) below the 7% forward error correction (FEC) limit of 3.8 × 10 −3. The performance of TFDNet verifies the effectiveness of time-frequency image analysis which has been applied to tackle nonlinear distortions in UVLC system for the first time.

show abstract

“…By using various decoding strategies that were proven in this study, both speed and performance were improved without changes to the model structure. Performance may be further improved by integrating ideas proposed in other studies in which models were lightened through strategies such as network pruning [40], knowledge distillation [41], and quantization [15]. Hence, an effective beam search technique and a new decoding technique will be investigated in future studies.…”

Section: Discussionmentioning

confidence: 99%

Decoding Strategies for Improving Low-Resource Machine Translation

et al. 2020

View full text Add to dashboard Cite

Pre-processing and post-processing are significant aspects of natural language processing (NLP) application software. Pre-processing in neural machine translation (NMT) includes subword tokenization to alleviate the problem of unknown words, parallel corpus filtering that only filters data suitable for training, and data augmentation to ensure that the corpus contains sufficient content. Post-processing includes automatic post editing and the application of various strategies during decoding in the translation process. Most recent NLP researches are based on the Pretrain-Finetuning Approach (PFA). However, when small and medium-sized organizations with insufficient hardware attempt to provide NLP services, throughput and memory problems often occur. These difficulties increase when utilizing PFA to process low-resource languages, as PFA requires large amounts of data, and the data for low-resource languages are often insufficient. Utilizing the current research premise that NMT model performance can be enhanced through various pre-processing and post-processing strategies without changing the model, we applied various decoding strategies to Korean–English NMT, which relies on a low-resource language pair. Through comparative experiments, we proved that translation performance could be enhanced without changes to the model. We experimentally examined how performance changed in response to beam size changes and n-gram blocking, and whether performance was enhanced when a length penalty was applied. The results showed that various decoding strategies enhance the performance and compare well with previous Korean–English NMT approaches. Therefore, the proposed methodology can improve the performance of NMT models, without the use of PFA; this presents a new perspective for improving machine translation performance.

show abstract

Importance Estimation for Neural Network Pruning

Cited by 665 publications

References 22 publications

Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks

Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks

Nonlinear Resilient Learning Method Based on Joint Time-Frequency Image Analysis in Underwater Visible Light Communication

Decoding Strategies for Improving Low-Resource Machine Translation

Contact Info

Product

Resources

About