Advancing RNN Transducer Technology for Speech Recognition

Abstract

We investigate a set of techniques for RNN Transducers (RNN-Ts) that were instrumental in lowering the word error rate on three different tasks (Switchboard 300 hours, conversational Spanish 780 hours and conversational Italian 900 hours). The techniques pertain to architectural changes, speaker adaptation, language model fusion, model combination and general training recipe. First, we introduce a novel multiplicative integration of the encoder and prediction network vectors in the joint network (as opposed to additive). Second, we discuss the applicability of i-vector speaker adaptation to RNN-Ts in conjunction with data perturbation. Third, we explore the effectiveness of the recently proposed density ratio language model fusion for these tasks. Last but not least, we describe the other components of our training recipe and their effect on recognition performance. We report a 5.9% and 12.5% word error rate on the Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation and a 12.7% WER on the Mozilla CommonVoice Italian test set.

4-bit Quantization of LSTM-based Speech Recognition Models

“…The prediction network is a single unidirectional LSTM layer with 768 cells. Encoder and prediction network outputs are combined multiplicatively in a joint network [3], with an FC layer and log-Softmax over 46 output characters. We train for 20 epochs with batch size 64, using AdamW and a triangular LR policy (OneCy-cleLR), on the audio and character-level transcripts from the SWB corpus, augmented with speed and tempo perturbation [21], SpecAugment [22], and Sequence Noise Injection [23].…”

“…In the acoustic encoder, all LSTM layers beside the first are quantized to 4 bits; this dramatically increases their computation throughput and reduces encoder runtime by 2.6× (blue bars). Due to the iterative beam search (decoding) process [3], the decoder runtime increases significantly with beam width. Thanks to the quantized prediction network, the decoding time (red bars) scales well between FP16 and INT4, achieving 3.3× speed-up, mitigating the impact of wider beams.…”

“…Overall, the INT4 implementation achieves 2.6× speed-up compared to FP16 across different beam widths. 3…”

“…Neural Network (NN) models for ASR have achieved tremendous success in the past decade, significantly reducing the gap with human performance [1,2]. Long Short-Term Memory (LSTM) building blocks are at the core of ASR models, either in NN-HMM hybrid [1] or NN end-to-end form [2,3], delivering state-of-the-art performance on standard speech benchmarks, such as the SWB corpus [4].…”

“…This decisively mitigates the degradation (grey line). (2) We apply QAT to fine-tune a state-of-the-art pre-trained model (red line)1 ; it has been shown that starting from a pre-trained model requires significantly fewer train iterations[28]; this is particularly important for RNN-T.(3) Instead of OneCycleLR[3], we use a custom LR policy (see Sec. 3.3 and Fig.3(d)) which achieves better performance (purple line).…”