ShrinkML: End-to-End ASR Model Compression Using Reinforcement Learning

Abstract

End-to-end automatic speech recognition (ASR) models are increasingly large and complex to achieve the best possible accuracy. In this paper, we build an AutoML system that uses reinforcement learning (RL) to optimize the per-layer compression ratios when applied to a state-of-the-art attention based end-to-end ASR model composed of several LSTM layers. We use singular value decomposition (SVD) low-rank matrix factorization as the compression method. For our RL-based Au-toML system, we focus on practical considerations such as the choice of the reward/punishment functions, the formation of an effective search space, and the creation of a representative but small data set for quick evaluation between search steps. Finally, we present accuracy results on LibriSpeech of the model compressed by our AutoML system, and we compare it to manually-compressed models. Our results show that in the absence of retraining our RL-based search is an effective and practical method to compress a production-grade ASR system. When retraining is possible, we show that our AutoML system can select better highly-compressed seed models compared to manually hand-crafted rank selection, thus allowing for more compression than previously possible.

FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

“…In this manner, we remove both the excessive overhead of fine-tuning and the need for labelled data availability, which is crucial for real-world, privacy-aware applications (Wainwright et al, 2012;Shokri & Shmatikov, 2015). Finally, other model compression methods (Fang et al, 2018;Wang et al, 2019a;Dudziak et al, 2019) remain orthogonal to FjORD. System Heterogeneity.…”

Mutual-learning sequence-level knowledge distillation for automatic speech recognition

“…NAS aims at automatically designing neural network architectures [15][16][17][18]. For example, He et al [17] proposed the AutoML for Model Compression (AMC) method and introduced reinforcement learning to learn the optimal parameters of pruning; Dudziak et al [18] used reinforcement learning to select the per-layer compression ratios based on matrix approximation. However, such a method requires massive computation during the search.…”

Journey Towards Tiny Perceptual Super-Resolution

“…Neural Architecture Search for Super-resolution. Recent SR works aim to build more efficient models using NAS, which has been vastly successful in a wide range of tasks such as image classification [57,55,38], language modeling [56], and automatic speech recognition [12]. We mainly focus on previous works that adopt NAS for SR and refer the reader to Elsken et al [13] for a detailed survey on NAS.…”