Online Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture

Abstract

Recently, there has been increasing progress in endto-end automatic speech recognition (ASR) architecture, which transcribes speech to text without any pre-trained alignments. One popular end-to-end approach is the hybrid Connectionist Temporal Classification (CTC) and attention (CTC/attention) based ASR architecture, which utilizes the advantages of both CTC and attention. The hybrid CTC/attention ASR systems exhibit performance comparable to that of the conventional deep neural network (DNN) / hidden Markov model (HMM) ASR systems. However, how to deploy hybrid CTC/attention systems for online speech recognition is still a non-trivial problem. This paper describes our proposed online hybrid CTC/attention endto-end ASR architecture, which replaces all the offline components of conventional CTC/attention ASR architecture with their corresponding streaming components. Firstly, we propose stable monotonic chunk-wise attention (sMoChA) to stream the conventional global attention, and further propose monotonic truncated attention (MTA) to simplify sMoChA and solve the training-and-decoding mismatch problem of sMoChA. Secondly, we propose truncated CTC (T-CTC) prefix score to stream CTC prefix score calculation. Thirdly, we design dynamic waiting joint decoding (DWJD) algorithm to dynamically collect the predictions of CTC and attention in an online manner. Finally, we use latency-controlled bidirectional long short-term memory (LC-BLSTM) to stream the widely-used offline bidirectional encoder network. Experiments with LibriSpeech English and HKUST Mandarin tasks demonstrate that, compared with the offline CTC/attention model, our proposed online CTC/attention model improves the real time factor in human-computer interaction services and maintains its performance with moderate degradation. To the best of our knowledge, this is the first work to provide the full-scale online solution for CTC/attention end-to-end ASR architecture.

Automatic Speech Recognition: Systematic Literature Review

“…In [47], the authors proposed an online hybrid CTC/attention E2E ASR architecture that replaces all the offline components of a conventional CTC/attention ASR architecture with their corresponding streaming components by using LibriSpeech English and Mandarin tasks (from the Hong Kong University of Science and Technology, HKUST) to decode the speech in a low-latency and real-time manner. The researchers in [92] introduced a combined framework to integrate social signal detection (SSD) and ASR systems based on CTC, which is an end-to-end model.…”

Preventing Early Endpointing for Online Automatic Speech Recognition

“…MTA [19] aims to solve the training and decoding mismatch problem. Specifically, MoChA and sMoChA only take the context with a predefined chunk width w during decoding, but they receive the full historical information of input during training.…”

“…7. For full details, please refer to [19]. We use both sMoChA and MTA techniques in the experiments section.…”

End-to-end Speech Recognition with Adaptive Computation Steps

“…This not only speeds up the model training, but also expands the scope of attention to all the encoding timesteps before the current truncating point. In addition, MTA has so far given the best ASR performance among various hard attention mechanisms according to [28].…”