Attention Based On-Device Streaming Speech Recognition with Large Speech Corpus

Abstract

In this paper, we present a new on-device automatic speech recognition (ASR) system based on monotonic chunk-wise attention (MoChA) models trained with large (> 10K hours) corpus. We attained around 90% of a word recognition rate for general domain mainly by using joint training of connectionist temporal classifier (CTC) and cross entropy (CE) losses, minimum word error rate (MWER) training, layer-wise pretraining and data augmentation methods. In addition, we compressed our models by more than 3.4 times smaller using an iterative hyper low-rank approximation (LRA) method while minimizing the degradation in recognition accuracy. The memory footprint was further reduced with 8-bit quantization to bring down the final model size to lower than 39 MB. For on-demand adaptation, we fused the MoChA models with statistical n-gram models, and we could achieve a relatively 36% improvement on average in word error rate (WER) for target domains including the general domain.

Small Energy Masking for Improved Neural Network Training for End-To-End Speech Recognition

“…As shown in Fig. 2, we apply the "global" mean and variance normalization as in [17], since the utterance-by-utterance mean and variance normalizations are not easily realizable for streaming speech recognition [18]. Note that mean subtraction must be applied before masking, otherwise, the non-zero values in the masked region will distort the model during the training.…”

End-to-End Training of a Large Vocabulary End-to-End Speech Recognition System

“…We have tried various types of training strategies for better performance [53,54]. Our MoCha implementation and optimization are described in very detail in our another paper [50]. The structure of our entire end-to-end speech recognition system is shown in Fig.…”

ShrinkML: End-to-End ASR Model Compression Using Reinforcement Learning

“…We use an end-to-end attention based ASR model [11,12] with an architecture similar to the one proposed in [10] as depicted in Fig. 1.…”

“…t . c o h o r t m o d e l s 2 f o r u t t e r a n c e i n v a l i d a t i o n s e t s : c o r r e l a t i o n ( w e r u t t , w e r a v g a l l ) 9 10 # f i l t e r u t t e r a n c e s < m i n c o r r e l and m i n l e n g t h 11 f o r u t t e r a n c e i n v a l i d a t i o n s e t s : 12 i f c o r r e l [ u t t e r a n c e ] > c o r r e l m i n : 13 n e w s e t . a p p e n d ( s a m p l e ) Listing 1: Heuristic to find "condensed" datasets in Fig.…”