Joint CTC-attention based end-to-end speech recognition using multi-task learning

Abstract

Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments. One approach is the attention-based encoderdecoder framework that learns a mapping between variable-length input and output sequences in one step using a purely data-driven method. The attention model has often been shown to improve the performance over another end-to-end approach, the Connectionist Temporal Classification (CTC), mainly because it explicitly uses the history of the target character without any conditional independence assumptions. However, we observed that the performance of the attention has shown poor results in noisy condition and is hard to learn in the initial training stage with long input sequences. This is because the attention model is too flexible to predict proper alignments in such cases due to the lack of left-to-right constraints as used in CTC. This paper presents a novel method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4-14.6% relative improvements in Character Error Rate (CER).

A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

“…End-to-end (E2E) models [2,3,4,5,6,7,8,9] have gained large popularity in the automatic speech recognition (ASR) community over the last few years. These models replace components of a conventional ASR system, namely an acoustic (AM), pronunciation (PM) and language models (LM), with a single neural network.…”

End-To-End Multi-Speaker Speech Recognition With Transformer

“…In this subsection, we briefly introduce the end-to-end singlechannel multi-speaker speech recognition model proposed in [8,9], shown in Fig.1. The model is an extension of the joint CTC/attentionbased encoder-decoder framework [14] to recognize multi-speaker speech. The input O = {o1, .…”

“…For multi-channel multi-speaker speech recognition, an end-to-end system was proposed in [13], called MIMO-Speech because of the multi-channel input (MI) and multi-speaker output (MO). This system consists of a maskbased neural beamformer frontend, which explicitly separates the multi-speaker speech via beamforming, and an end-to-end speech recognition model backend based on the joint CTC/attention-based encoder-decoder [14] to recognize the separated speech streams. This end-to-end architecture is optimized via only the connectionist temporal classification (CTC) and cross-entropy (CE) losses in the backend ASR, but is nonetheless able to learn to develop relatively good separation abilities.…”

Attention Based On-Device Streaming Speech Recognition with Large Speech Corpus

“…A CTC loss can be readily applicable for training MoChA Model, especially the encoder, because it also leads the alignment between input and output as a monotonic manner. Moreover CTC loss has the advantage of learning alignment in noisy environments and can help to quickly learn the alignment of the attention based model through joint training [18].…”