Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict

Abstract

We present Mask CTC, a novel non-autoregressive end-to-end automatic speech recognition (ASR) framework, which generates a sequence by refining outputs of the connectionist temporal classification (CTC). Neural sequence-to-sequence models are usually autoregressive: each output token is generated by conditioning on previously generated tokens, at the cost of requiring as many iterations as the output length. On the other hand, non-autoregressive models can simultaneously generate tokens within a constant number of iterations, which results in significant inference time reduction and better suits end-toend ASR model for real-world scenarios. In this work, Mask CTC model is trained using a Transformer encoder-decoder with joint training of mask prediction and CTC. During inference, the target sequence is initialized with the greedy CTC outputs and low-confidence tokens are masked based on the CTC probabilities. Based on the conditional dependence between output tokens, these masked low-confidence tokens are then predicted conditioning on the high-confidence tokens. Experimental results on different speech recognition tasks show that Mask CTC outperforms the standard CTC model (e.g., 17.9% → 12.1% WER on WSJ) and approaches the autoregressive model, requiring much less inference time using CPUs (0.07 RTF in Python implementation). All of our codes will be publicly available.

Improved Mask-CTC for Non-Autoregressive End-to-End ASR

“…Due to the conditional independence assumption, the CTC-based model generally suffers from poor recognition performance [6]. Mask-CTC has been proposed to mitigate this problem by iteratively refining an output of CTC with bi-directional contexts of tokens [29]. Mask-CTC adopts an encoder-decoder model built upon Transformer blocks [16].…”

“…NAR models in neural machine translation have achieved competitive performance to AR models [18][19][20][21][22][23][24][25]. Several approaches have been proposed to realize a NAR model in E2E-ASR [26][27][28][29][30][31]. Connectionist temporal classification (CTC) predicts a frame-wise latent alignment between input speech frames and output tokens, and generates a target sequence based on a conditional independence assumption between the frame predictions [26].…”

“…Despite achieving comparable performance with that of AR models, Imputer processes the frame-level sequence, consisting of hundreds of units, using the self-attention layers [16], which cost computations proportional to the square of a sequence length. On the other hand, Mask-CTC [29] generates a sequence shorter than Imputer by refining a token-level CTC output with the mask prediction, achieving fast inference speed of less than 0.1 real time factor (RTF) using CPU.…”

CASS-NAT: CTC Alignment-Based Single Step Non-Autoregressive Transformer for Speech Recognition

“…Prevailing iterative NATs regard the decoder as a masked language model, where the tokens with low confidence are first masked and then new tokens are predicted from unmasked tokens. The two steps are conducted alternately within a constant number of iterations [13][14][15]. Length prediction of the decoder input is difficult for NAT models.…”

“…Length prediction of the decoder input is difficult for NAT models. To address the issue, Higuchi et al used the length of recognition results of CTC [14], while Chan et al proposed a model named as Imputer, which directly used the length of input feature sequence [16]. Compared to the left-to-right generation order in an autoregressive transformer (AT), an iterative NAT essentially adopts a different token generation order, where the iterations are still required.…”