Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration

Abstract

The state-of-the-art neural network architecture named Transformer has been used successfully for many sequence-tosequence transformation tasks. The advantage of this architecture is that it has a fast iteration speed in the training stage because there is no sequential operation as with recurrent neural networks (RNN). However, an RNN is still the best option for end-to-end automatic speech recognition (ASR) tasks in terms of overall training speed (i.e., convergence) and word error rate (WER) because of effective joint training and decoding methods. To realize a faster and more accurate ASR system, we combine Transformer and the advances in RNN-based ASR. In our experiments, we found that the training of Transformer is slower than that of RNN as regards the learning curve and integration with the naive language model (LM) is difficult. To address these problems, we integrate connectionist temporal classification (CTC) with Transformer for joint training and decoding. This approach makes training faster than with RNNs and assists LM integration. Our proposed ASR system realizes significant improvements in various ASR tasks. For example, it reduced the WERs from 11.1% to 4.5% on the Wall Street Journal and from 16.1% to 11.6% on the TED-LIUM by introducing CTC and LM integration into the Transformer baseline.

Streaming Automatic Speech Recognition with the Transformer Model

“…An RNN-based language model (LM) is employed via shallow fusion. The RNN-LM consists of 4 LSTM layers with 2048 units each [13], CTC prefix beam search decoding only [20], and attention beam search decoding only [3]. In addition, results for including the RNN-LM, for using data augmentation [25] as well as for the large transformer setup are shown.…”

“…In addition, results for including the RNN-LM, for using data augmentation [25] as well as for the large transformer setup are shown. Table 1 presents ASR results of our transformer-based baseline systems, which are jointly trained with CTC to optimize training convergence and ASR accuracy [3,13]. Results of different decoding methods are shown with and without using the RNN-LM, SpecAugment [25], and the large transformer model.…”

“…Hybrid hidden Markov model (HMM) based automatic speech recognition (ASR) systems have provided state-of-the-art results for the last few decades [1,2]. End-to-end ASR systems, which approach the speech-to-text conversion problem using a single sequence-to-sequence model, have recently demonstrated competitive performance [3]. The most popular and successful end-to-end ASR approaches are based on connectionist temporal classification (CTC) [4], recurrent neural network (RNN) transducer (RNN-T) [5], and attention-based encoder-decoder architectures [6].…”

Transformer-Based Online CTC/Attention End-To-End Speech Recognition Architecture

“…Recently, Transformer [12] has gained success in ASR field [13,14,15]. Transformer-based models are parallelizable and competitive to recurrent neural networks [16].…”

End-To-End Multi-Speaker Speech Recognition With Transformer

“…Recently, Transformer models [15] have shown impressive performance in many tasks, such as pretrained language models [16,17], end-to-end speech recognition [18,19], and speaker diarization [20], surpassing the long short-term memory recurrent neural networks (LSTM-RNNs) based models. One of the key components in the Transformer model is self-attention, which computes the contribution information of the whole input sequence and maps the sequence into a vector at every time step.…”