Model Compression Applied to Small-Footprint Keyword Spotting

Abstract

Several consumer speech devices feature voice interfaces that perform on-device keyword spotting to initiate user interactions. Accurate on-device keyword spotting within a tight CPU budget is crucial for such devices. Motivated by this, we investigated two ways to improve deep neural network (DNN) acoustic models for keyword spotting without increasing CPU usage. First, we used low-rank weight matrices throughout the DNN. This allowed us to increase representational power by increasing the number of hidden nodes per layer without changing the total number of multiplications. Second, we used knowledge distilled from an ensemble of much larger DNNs used only during training. We systematically evaluated these two approaches on a massive corpus of far-field utterances. Alone both techniques improve performance and together they combine to give significant reductions in false alarms and misses without increasing CPU or memory usage.

Improving Noise Robustness of Automatic Speech Recognition via Parallel Data and Teacher-student Learning

“…We focus on improving the ASR performance under multimedia noise which is commonly present at home. T/S learning was at first explored in speech community [22] [23] to distill the knowledge from bigger models to a smaller one, and was successfully applied in the areas of ASR [24,25] and keyword spotting [26] afterwards. Instead of knowledge distillation, we adopt the T/S learning for domain adaptation which was proposed in [27] to build an ASR system performing more robustly under multimedia noise.…”

End-to-end Streaming Keyword Spotting

“…our first stage, always-on network has about 13K parameters [8]). The SVDF also pairs very well with linear bottleneck layers to significantly reduce the param- eter count as in [18,19], and more recently in [10]. And because it allows for creating evenly sized deep networks, we can insert them throughout the network as in Figure 3.…”

Multi-Task Learning and Weighted Cross-Entropy for DNN-Based Keyword Spotting

“…Keyword (KW) spotting in continuous speech has been an area of research for more than two decades [14]- [23]. In much of recent work, latency and computation are not concerns, and offline large vocabulary speech recognition systems can be used to decode the audio utterances and create transcripts or lattices of words and/or phones which can then be searched for the presence of the keyword(s) of interest [14]- [16].…”

“…The background model is also sometimes called the filler or garbage model, and may be a simple speech/non-speech loop HMM [18], or may involve a loop over phones or words [21]. With the growing success of deep learning in recent years, novel techniques using Deep Neural Networks (DNNs) and Recurrent Neural Networks (RNNs) that do not involve HMMs have also been proposed [22,6,23].…”