RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition

Dong, Peiyan; Wang, Siyue; Niu, Wei; Zhang, Chengming; Lin, Sheng; Li, Zhengang; Gong, Yifan; Ren, Bin; Lin, Xue; Wang, Yanzhi; Tao, Dingwen

doi:10.48550/arxiv.2002.11474

Cited by 6 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This leads to irregular weight distribution and inevitably introduces extra indices to store the locations of pruned weights. Eventually, this drawback limits performance acceleration [8].…”

Section: Background 21 Dnn Model Pruningmentioning

confidence: 99%

“…Pruning Algorithms. There are three main issues of the existing pattern-based pruning algorithms [8,35]: (1) The existing algorithms select pattern for each kernel via estimating weight importance based on magnitude. However, this estimation approach requires a well-trained model, whose weights will not change dramatically, which is not true for training from scratch.…”

Section: Issues Of Existing Patternmentioning

confidence: 99%

“…Optimization. The computation efficiency optimizations proposed by existing works [8,35] for patternbased pruning cannot be directly applied to our training framework. This is mainly because of two reasons.…”

Section: Lack Of Training Efficiencymentioning

confidence: 99%

“…This is mainly because of two reasons. On one hand, the existing pattern-based acceleration techniques [8,35] are designed for accelerating inference (i.e., forward phase) instead of training (including both forward and back phases). However, based on our profiling result as shown in Figure 3, backward phase can consume more than 70% of the overall training FLOPs.…”

Section: Lack Of Training Efficiencymentioning

confidence: 99%

See 3 more Smart Citations

ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning

Zhang,

Yuan,

Niu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are becoming increasingly deeper, wider, and non-linear because of the growing demand on prediction accuracy and analysis quality. The wide and deep CNNs, however, require a large amount of computing resources and processing time. Many previous works have studied model pruning to improve inference performance, but little work has been done for effectively reducing training cost. In this paper, we propose Click-Train: an efficient and accurate end-to-end training and pruning framework for CNNs. Different from the existing pruning-duringtraining work, ClickTrain provides higher model accuracy and compression ratio via fine-grained architecture-preserving pruning. By leveraging pattern-based pruning with our proposed novel accurate weight importance estimation, dynamic pattern generation and selection, and compiler-assisted computation optimizations, ClickTrain generates highly accurate and fast pruned CNN models for direct deployment without any time overhead, compared with the baseline training. ClickTrain also reduces the end-to-end time cost of the state-of-the-art pruning-after-training methods by up to about 67% with comparable accuracy and compression ratio. Moreover, compared with the state-of-the-art pruning-during-training approach, ClickTrain reduces the accuracy drop by up to 2.1% and improves the compression ratio by up to 2.2× on the tested datasets, under similar limited training time.

show abstract

Section: Background 21 Dnn Model Pruningmentioning

confidence: 99%

Section: Issues Of Existing Patternmentioning

confidence: 99%

Section: Lack Of Training Efficiencymentioning

confidence: 99%

Section: Lack Of Training Efficiencymentioning

confidence: 99%

See 2 more Smart Citations

ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning

Zhang,

Yuan,

Niu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Different from the prior work on coarse-grained pruning and NAS that find a smaller, yet regular, DNN structure, recent work [16], [48], [58] propose to prune the weights in a more fine-grained manner, e.g., assigning potentially different patterns to kernels. Higher accuracy can be achieved as a result of the intra-kernel flexibility, while high hardware parallelism (and mobile inference acceleration) can be achieved with the assist of compiler-level code generation techniques [58].…”

Section: Introductionmentioning

confidence: 99%

NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration

Li¹,

Yuan²,

Niu³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

With the increasing demand to efficiently deploy DNNs on mobile edge devices, it becomes much more important to reduce unnecessary computation and increase the execution speed. Prior methods towards this goal, including model compression and network architecture search (NAS), are largely performed independently, and do not fully consider compilerlevel optimizations which is a must-do for mobile acceleration. In this work, we first propose (i) a general category of fine-grained structured pruning applicable to various DNN layers, and (ii) a comprehensive, compiler automatic code generation framework supporting different DNNs and different pruning schemes, which bridge the gap of model compression and NAS. We further propose NPAS, a compiler-aware unified network pruning and architecture search. To deal with large search space, we propose a meta-modeling procedure based on reinforcement learning with fast evaluation and Bayesian optimization, ensuring the total number of training epochs comparable with representative NAS frameworks. Our framework achieves 6.7ms, 5.9ms, and 3.9ms ImageNet inference times with 78.2%, 75% (MobileNet-V3 level), and 71% (MobileNet-V2 level) Top-1 accuracy respectively on an off-the-shelf mobile phone, consistently outperforming prior work.

show abstract

Online Speech Recognition Using Multichannel Parallel Acoustic Score Computation and Deep Neural Network (DNN)- Based Voice-Activity Detector

Park

2020

Applied Sciences

View full text Add to dashboard Cite

This paper aims to design an online, low-latency, and high-performance speech recognition system using a bidirectional long short-term memory (BLSTM) acoustic model. To achieve this, we adopt a server-client model and a context-sensitive-chunk-based approach. The speech recognition server manages a main thread and a decoder thread for each client and one worker thread. The main thread communicates with the connected client, extracts speech features, and buffers the features. The decoder thread performs speech recognition, including the proposed multichannel parallel acoustic score computation of a BLSTM acoustic model, the proposed deep neural network-based voice activity detector, and Viterbi decoding. The proposed acoustic score computation method estimates the acoustic scores of a context-sensitive-chunk BLSTM acoustic model for the batched speech features from concurrent clients, using the worker thread. The proposed deep neural network-based voice activity detector detects short pauses in the utterance to reduce response latency, while the user utters long sentences. From the experiments of Korean speech recognition, the number of concurrent clients is increased from 22 to 44 using the proposed acoustic score computation. When combined with the frame skipping method, the number is further increased up to 59 clients with a small accuracy degradation. Moreover, the average user-perceived latency is reduced from 11.71 s to 3.09–5.41 s by using the proposed deep neural network-based voice activity detector.

show abstract

RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition

Cited by 6 publications

References 28 publications

ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning

ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning

NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration

Online Speech Recognition Using Multichannel Parallel Acoustic Score Computation and Deep Neural Network (DNN)- Based Voice-Activity Detector

Contact Info

Product

Resources

About