Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment

Chi, Ethan A.; Salazar, Julián; Kirchhoff, Katrin

doi:10.48550/arxiv.2010.14233

Cited by 2 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there is a disadvantage of the AR models in that the inference time linearly increases with the output length. Recently, nonautoregressive (NAR) models has gained more and more attention in sequence-to-sequence tasks, including machine translation [5][6][7], speech recognition (ASR) [1,[8][9][10][11], and speech translation [12]. In contrast to the AR modeling, NAR modeling can predict the output tokens concurrently, the inference speed of which is dramatically faster than AR modeling.…”

Section: Introductionmentioning

confidence: 99%

Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models

Wang¹,

Fujita²,

Chang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. With recent state-of-the-art attention-based automatic speech recognition (ASR) structure, NAR can realize promising real-time factor (RTF) improvement with only small degradation of accuracy compared to the autoregressive (AR) models. However, the recognition inference needs to wait for the completion of a full speech utterance, which limits their applications on low latency scenarios. To address this issue, we propose a novel end-to-end streaming NAR speech recognition system by combining blockwiseattention and connectionist temporal classification with maskpredict (Mask-CTC) NAR. During inference, the input audio is separated into small blocks and then processed in a blockwise streaming way. To address the insertion and deletion error at the edge of the output of each block, we apply an overlapping decoding strategy with a dynamic mapping trick that can produce more coherent sentences. Experimental results show that the proposed method improves online ASR recognition in low latency conditions compared to vanilla Mask-CTC. Moreover, it can achieve a much faster inference speed compared to the AR attention-based models. All of our codes will be publicly available at https://github.com/espnet/espnet.

show abstract

Section: Introductionmentioning

confidence: 99%

Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models

Wang¹,

Fujita²,

Chang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To accelerate the inference, non-autoregressive transformers (NAT) were proposed for the parallel generation of the output sequence. The idea is widely adopted in neural machine translation (NMT) [4][5][6], automatic speech recognition (ASR) [7][8][9][10][11][12][13][14][15][16][17][18], text-to-speech (TTS) [19,20] and speech translation [21].…”

Section: Introductionmentioning

confidence: 99%

“…In addition, Fujita et al used the idea of the insertion transformer from NMT to generate the output sequence with an arbitrary order [12]. Another recent effective method is using multiple decoders as refiners to do an iterative refinement based on CTC alignments [14]. Theoretically, the iterative NAT has a limited improvement of inference speed since multiple iterations are still needed to obtain a competitive result.…”

Section: Introductionmentioning

confidence: 99%

An Improved Single Step Non-Autoregressive Transformer for Automatic Speech Recognition

Fan

Chu²,

Chang³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

Non-autoregressive mechanisms can significantly decrease inference time for speech transformers, especially when the single step variant is applied. Previous work on CTC alignmentbased single step non-autoregressive transformer (CASS-NAT) has shown a large real time factor (RTF) improvement over autoregressive transformers (AT). In this work, we propose several methods to improve the accuracy of the end-to-end CASS-NAT, followed by performance analyses. First, convolution augmented self-attention blocks are applied to both the encoder and decoder modules. Second, we propose to expand the trigger mask (acoustic boundary) for each token to increase the robustness of CTC alignments. In addition, iterated loss functions are used to enhance the gradient update of low-layer parameters. Without using an external language model, the WERs of the improved CASS-NAT, when using the three methods, are 3.1%/7.2% on Librispeech test clean/other sets and the CER is 5.4% on the Aishell1 test set, achieving a 7%∼21% relative WER/CER improvement. For the analyses, we plot attention weight distributions in the decoders to visualize the relationships between token-level acoustic embeddings. When the acoustic embeddings are visualized, we find that they have a similar behavior to word embeddings, which explains why the improved CASS-NAT performs similarly to AT.

show abstract

Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment

Cited by 2 publications

References 9 publications

Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models

Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models

An Improved Single Step Non-Autoregressive Transformer for Automatic Speech Recognition

Contact Info

Product

Resources

About