Exploring Transformers for Large-Scale Speech Recognition

Abstract

While recurrent neural networks still largely define state-of-theart speech recognition systems, the Transformer network has been proven to be a competitive alternative, especially in the offline condition. Most studies with Transformers have been constrained in a relatively small scale setting, and some forms of data argumentation approaches are usually applied to combat the data sparsity issue. In this paper, we aim at understanding the behaviors of Transformers in the large-scale speech recognition setting, where we have used around 65,000 hours of training data. We investigated various aspects on scaling up Transformers, including model initialization, warmup training as well as different Layer Normalization strategies. In the streaming condition, we compared the widely used attention mask based future context lookahead approach to the Transformer-XL network. From our experiments, we show that Transformers can achieve around 6% relative word error rate (WER) reduction compared to the BLSTM baseline in the offline fashion, while in the streaming fashion, Transformer-XL is comparable to LC-BLSTM with 800 millisecond latency constraint.

Wake Word Detection with Streaming Transformers

“…When absolute embedding is used, FRR=1.1% at the same FAH. This contradicts the observations in [21,22] where same-layer dependency was found to be more advantageous for ASR and it was attributed to the fact that the receptive field is maximized at every layer 8 . A better way of incorporating relative positional information for this case is our future work.…”

“…The time and space complexity are both reduced to O(T ), and the within-chunk computation across time can be parallelized with GPUs. While there has been recent work [18,19,20,21,22] with similar ideas showing that such streaming Transformers achieve competitive performance compared with latency-controlled BiLSTMs [23] or non-streaming Transformers for ASR, it remains unclear how the streaming transformers work for shorter sequence modeling task like wake word detection.…”

Emformer: Efficient Memory Transformer Based Acoustic Model for Low Latency Streaming Speech Recognition

“…First, Emformer removes the duplicated computation from the left context block by caching the key and value in previous segments' self-attention. Second, rather than passing the memory bank within the current layer in AM-TRF, inspired by transformer-xl [2] and its applicatin in speech recognition [20], Emformer carries over the memory bank from the lower layer. Third, Emformer disables the summary vector's attention with memory bank to avoid overweighting the most left part of context information.…”

End-to-End Multi-Channel Transformer for Speech Recognition

“…Transformers [21] are powerful neural architectures that lately have been used in ASR [22][23][24], SLU [25], and other audio-visual applications [26] with great success, mainly due to their attention mechanism. Only until recently, the attention concept has also been applied to beamforming, specifically for speech and noise mask estimations [9,27].…”