Non-Autoregressive Transformer for Speech Recognition

Chen, Nanxin; Watanabe, Shinji; Villalba, Jesús; Żelasko, Piotr; Dehak, Najim

doi:10.1109/lsp.2020.3044547

Cited by 72 publications

(56 citation statements)

References 19 publications

(23 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance gains are higher when both ASR and MT models use the target units of the same granularity (BPE in our case). As all the models are transformers, they could be replaced with Non-Auto regressive models [19], [20], [21], which could reduce the latency of decoding. Along with the input symbols, a mechanism to present the confidence of the input symbol could help MT models to better translate the ASR hypothesis.…”

Section: Conclusion and Future Scopementioning

confidence: 99%

Jointly Trained Transformers Models for Spoken Language Translation

Vydana

Karafiát

Žmolíková

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

End-to-End and cascade (ASR-MT) spoken language translation (SLT) systems are reaching comparable performances, however, a large degradation is observed when translating the ASR hypothesis in comparison to using oracle input text. In this work, degradation in performance is reduced by creating an End-to-End differentiable pipeline between the ASR and MT systems. In this work, we train SLT systems with ASR objective as an auxiliary loss and both the networks are connected through the neural hidden representations. This training has an End-to-End differentiable path with respect to the final objective function and utilizes the ASR objective for better optimization. This architecture has improved the BLEU score from 41.21 to 44.69. Ensembling the proposed architecture with independently trained ASR and MT systems further improved the BLEU score from 44.69 to 46.9. All the experiments are reported on English-Portuguese speech translation task using the How2 corpus. The final BLEU score is on-par with the best speech translation system on How2 dataset without using any additional training data and language model and using fewer parameters.

show abstract

Section: Conclusion and Future Scopementioning

confidence: 99%

Jointly Trained Transformers Models for Spoken Language Translation

Vydana

Karafiát

Žmolíková

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Transformers rely on a simple but powerful procedure called attention, which focuses on certain parts of the input to get more efficient results. Currently, they are considered state-of-the-art models in sequential data, in particular natural language processing (NLP) methods such as machine translation [36], language modeling [37], and speech recognition [38]. The architecture of the Transformer developed by Vaswani et al [39] is based on the encoder-decoder model, which transforms a given sequence of elements into another sequence.…”

Section: Introductionmentioning

confidence: 99%

Vision Transformers for Remote Sensing Image Classification

et al. 2021

View full text Add to dashboard Cite

In this paper, we propose a remote-sensing scene-classification method based on vision transformers. These types of networks, which are now recognized as state-of-the-art models in natural language processing, do not rely on convolution layers as in standard convolutional neural networks (CNNs). Instead, they use multihead attention mechanisms as the main building block to derive long-range contextual relation between pixels in images. In a first step, the images under analysis are divided into patches, then converted to sequence by flattening and embedding. To keep information about the position, embedding position is added to these patches. Then, the resulting sequence is fed to several multihead attention layers for generating the final representation. At the classification stage, the first token sequence is fed to a softmax classification layer. To boost the classification performance, we explore several data augmentation strategies to generate additional data for training. Moreover, we show experimentally that we can compress the network by pruning half of the layers while keeping competing classification accuracies. Experimental results conducted on different remote-sensing image datasets demonstrate the promising capability of the model compared to state-of-the-art methods. Specifically, Vision Transformer obtains an average classification accuracy of 98.49%, 95.86%, 95.56% and 93.83% on Merced, AID, Optimal31 and NWPU datasets, respectively. While the compressed version obtained by removing half of the multihead attention layers yields 97.90%, 94.27%, 95.30% and 93.05%, respectively.

show abstract

“…To speed up the inference of speech recognition, nonautoregressive transformers have been proposed [5,6,7,8], which generate all target tokens simultaneously [5,6] or iteratively [7,8]. We notice that the encoder of AR transformers and that of NAR transformers are the same thus the dif- † Work performed during an internship at Tencent AI Lab.…”

Section: Introductionmentioning

confidence: 99%

“…For example, the NAR decoders in listen attentively and spell once (LASO) [5] and listen and fill in missing letter (LFML) [8] take fixed-length sequence filled with MASK tokens as input to predict target sequence:…”

Section: Introductionmentioning

confidence: 99%

Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input

Song

Huang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Non-autoregressive (NAR) transformer models have achieved significantly inference speedup but at the cost of inferior accuracy compared to autoregressive (AR) models in automatic speech recognition (ASR). Most of the NAR transformers take a fixed-length sequence filled with MASK tokens or a redundant sequence copied from encoder states as decoder input, they cannot provide efficient target-side information thus leading to accuracy degradation. To address this problem, we propose a CTC-enhanced NAR transformer, which generates target sequence by refining predictions of the CTC module. Experimental results show that our method outperforms all previous NAR counterparts and achieves 50x faster decoding speed than a strong AR baseline with only 0.0 ∼ 0.3 absolute CER degradation on Aishell-1 and Aishell-2 datasets.

show abstract

Non-Autoregressive Transformer for Speech Recognition

Cited by 72 publications

References 19 publications

Jointly Trained Transformers Models for Spoken Language Translation

Jointly Trained Transformers Models for Spoken Language Translation

Vision Transformers for Remote Sensing Image Classification

Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input

Contact Info

Product

Resources

About