Self-Distillation for Improving CTC-Transformer-Based ASR Systems

Moriya, Takafumi; Ochiai, Tsubasa; Karita, Shigeki; Satō, Hiroshi; Tanaka, Tomohiro; Ashihara, Takanori; Masumura, Ryo; Shinohara, Yusuke; Delcroix, Marc

doi:10.21437/interspeech.2020-1223

Cited by 16 publications

(13 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the LAS model, a WER reduction of 0.28% absolute (from 5.7% to 5.42%) on dev93 corresponds to a relative WER reduction of 7.4% on eval92, when using relaxed attention with γ = 0.1. For the transformer model, the best average result on dev93 is achieved with γ = 0.35, yielding an average WER of 5.80% on dev93 and 3.65% on eval92, which is an 18% relative improvement on eval92 compared to our own Baseline model (4.45%), and exceeds the current WSJ transformer state of the art by Moriya et al [25] (4.20%) by 13.1% relative, without adding any model complexity.…”

Section: Wall Street Journal (Wsj)mentioning

confidence: 55%

“…One effective method to deal with overconfidence in end-to-end ASR, introduced in [20], is label smoothing that blends the one-hot label with a uniform distribution or assigns part of the probability mass to tokens that are neighbors of the labeled token in the target sequence [14]. Interestingly, label smoothing is also effective against overfitting [21] and thus is commonly used for AED end-to-end ASR besides related regularization methods such as spectral augmentation [22], dropout [23], multi-task learning with an additional CTC loss [24,25], and the recently proposed multi-encoder-learning that uses additional encoders only during training [26]. Regularization methods applied to the crucial encoder-decoder attention mechanism in AED models were only recently discovered in [27], where CTC predictions in a multi-task learning setup are used to focus the attention in transformer models [27] to relevant frames in the encoded input sequence.…”

Section: Encoder Decodermentioning

confidence: 99%

See 1 more Smart Citation

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Lohrenz¹,

Schwarz²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, attention-based encoder-decoder (AED) models have shown high performance for end-to-end automatic speech recognition (ASR) across several tasks. Addressing overconfidence in such models, in this paper we introduce the concept of relaxed attention, which is a simple gradual injection of a uniform distribution to the encoder-decoder attention weights during training that is easily implemented with two lines of code. We investigate the effect of relaxed attention across different AED model architectures and two prominent ASR tasks, Wall Street Journal (WSJ) and Librispeech. We found that transformers trained with relaxed attention outperform the standard baseline models consistently during decoding with external language models. On WSJ, we set a new benchmark for transformer-based end-to-end speech recognition with a word error rate of 3.65%, outperforming state of the art (4.20%) by 13.1% relative, while introducing only a single hyperparameter. Upon acceptance, models will be published on github.

show abstract

Section: Wall Street Journal (Wsj)mentioning

confidence: 55%

Section: Encoder Decodermentioning

confidence: 99%

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Lohrenz¹,

Schwarz²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Combining both improved MEL-based models in the MEL-t-Fusion-Late approach yields the lowest WER of 3.40% on eval92, corresponding to a WER reduction of 0.91% absolute w.r.t. the normal Fusion-Late approach (4.31%), and a remarkable reduction of up to 19% relative compared to the best recently published transformer-based approach by Moriya et al [28], as shown in Table 2.…”

Section: Recognition Results and Discussionmentioning

confidence: 70%

“…Concerning the fusion of additional information into endto-end transformer models, the few existing approaches stem from audiovisual automatic speech recognition [24,25] and neural machine translation [26], where additional encoders are used to gather visual speech information or contextual information, respectively. Recent successful non-fusion techniques for end-to-end models are multi-task learning, e.g., by using a combination of CTC and attention-based losses [27,28], and augmentation techniques such as spectral augmentation [29]. Those methods improve neural networks by adding more variety to the trained models either by composite losses or by randomly withholding information in the input features.…”

Section: Introductionmentioning

confidence: 99%

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

Lohrenz¹,

Li²,

Fingscheidt³

2021

Preprint

View full text Add to dashboard Cite

Stream fusion, also known as system combination, is a common technique in automatic speech recognition for traditional hybrid hidden Markov model approaches, yet mostly unexplored for modern deep neural network end-to-end model architectures. Here, we investigate various fusion techniques for the all-attention-based encoder-decoder architecture known as the transformer, striving to achieve optimal fusion by investigating different fusion levels in an example single-microphone setting with fusion of standard magnitude and phase features. We introduce a novel multi-encoder learning method that performs a weighted combination of two encoder-decoder multi-head attention outputs only during training. Employing then only the magnitude feature encoder in inference, we are able to show consistent improvement on Wall Street Journal (WSJ) with language model and on Librispeech, without increase in runtime or parameters. Combining two such multi-encoder trained models by a simple late fusion in inference, we achieve state-of-the-art performance for transformer-based models on WSJ with a significant WER reduction of 19% relative compared to the current benchmark approach.

show abstract

“…Moriya et al distilled knowledge from a teacher AED model to a student CTC model [79]. Self-distillation in a single E2E ASR model has also been proposed as an in-place operation, from an offline mode to a streaming mode [61], [68], and from a Transformer decoder to a CTC layer [80]. Unlike those previous methods, we focus on distilling the positions of token boundaries learned in a CTC model to an AED model, rather than distilling the posterior distributions.…”

Section: Knowledge Distillation For Streaming Asrmentioning

confidence: 99%

Alignment Knowledge Distillation for Online Streaming Attention-Based Speech Recognition

Inaguma

Kawahara

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. AED models have achieved competitive performance in offline scenarios by jointly optimizing all components. They have recently been extended to an online streaming framework via models such as monotonic chunkwise attention (MoChA). However, the elaborate attention calculation process is not robust for long-form speech utterances. Moreover, the sequence-level training objective and time-restricted streaming encoder cause a nonnegligible delay in token emission during inference. To address these problems, we propose CTC synchronous training (CTC-ST), in which CTC alignments are leveraged as a reference for token boundaries to enable a MoChA model to learn optimal monotonic input-output alignments. We formulate a purely end-to-end training objective to synchronize the boundaries of MoChA to those of CTC. The CTC model shares an encoder with the MoChA model to enhance the encoder representation. Moreover, the proposed method provides alignment information learned in the CTC branch to the attention-based decoder. Therefore, CTC-ST can be regarded as self-distillation of alignment knowledge from CTC to MoChA. Experimental evaluations on a variety of benchmark datasets show that the proposed method significantly reduces recognition errors and emission latency simultaneously. The robustness to long-form and noisy speech is also demonstrated. We compare CTC-ST with several methods that distill alignment knowledge from a hybrid ASR system and show that the CTC-ST can achieve a comparable tradeoff of accuracy and latency without relying on external alignment information. The best MoChA system shows recognition accuracy comparable to that of RNNtransducer (RNN-T) while achieving lower emission latency.

show abstract

Self-Distillation for Improving CTC-Transformer-Based ASR Systems

Cited by 16 publications

References 24 publications

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition

Alignment Knowledge Distillation for Online Streaming Attention-Based Speech Recognition

Contact Info

Product

Resources

About