Alignment Knowledge Distillation for Online Streaming Attention-Based Speech Recognition

Abstract

This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. AED models have achieved competitive performance in offline scenarios by jointly optimizing all components. They have recently been extended to an online streaming framework via models such as monotonic chunkwise attention (MoChA). However, the elaborate attention calculation process is not robust for long-form speech utterances. Moreover, the sequence-level training objective and time-restricted streaming encoder cause a nonnegligible delay in token emission during inference. To address these problems, we propose CTC synchronous training (CTC-ST), in which CTC alignments are leveraged as a reference for token boundaries to enable a MoChA model to learn optimal monotonic input-output alignments. We formulate a purely end-to-end training objective to synchronize the boundaries of MoChA to those of CTC. The CTC model shares an encoder with the MoChA model to enhance the encoder representation. Moreover, the proposed method provides alignment information learned in the CTC branch to the attention-based decoder. Therefore, CTC-ST can be regarded as self-distillation of alignment knowledge from CTC to MoChA. Experimental evaluations on a variety of benchmark datasets show that the proposed method significantly reduces recognition errors and emission latency simultaneously. The robustness to long-form and noisy speech is also demonstrated. We compare CTC-ST with several methods that distill alignment knowledge from a hybrid ASR system and show that the CTC-ST can achieve a comparable tradeoff of accuracy and latency without relying on external alignment information. The best MoChA system shows recognition accuracy comparable to that of RNNtransducer (RNN-T) while achieving lower emission latency.

Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM

“…We also compared our method to knowledge distillation (KD) with MLM. To apply KD, we utilized a forced-aligned CTC path to align frame-level predictions of CTC with tokenlevel predictions of MLM, inspired by [40]. The KD loss function is formulated as…”

Distilling the Knowledge of BERT for Sequence-to-Sequence ASR

“…To solve the problem, we propose to use the forced alignment result, that is, the most plausible CTC path π = (π1, ..., πT ) of π ∈ B −1 (y) like [26], where the CTC path π is used to enable a monotonic chunkwise attention (MoChA) model to learn optimal alignments for streaming ASR. π can be obtained by tracking the path that has the maximum products of forward and backward variables, which can be obtained in the process of calculating the CTC loss function LCTC from Eq.…”

Knowledge Distillation-Based Training of Speech Enhancement for Noise-Robust Automatic Speech Recognition

“…Several studies have leveraged linguistic and acoustic information from pretrained ASR models to enhance ASR performance [35]- [37]. The KD approach using pretrained ASR models can be categorized into decoder-and encoderside methods [36].…”

“…The decoder-side KD method, which uses the output vector of the ASR decoder, transfers the global context to the student model, because the decoder output vector integrates the latent vectors from the ASR encoder with a large context window to capture the linguistic unit information. Thus, the decoder-side KD method maximizes transition probabilities between linguistic units, resulting in its inability to handle the frame-wise characteristics inherent in the ASR encoder [36], [37]. Conversely, the encoder-side KD method uses the output vector of the ASR encoder optimized in a frame-wise fashion with a fixed-size sliding window.…”