Encoder Transfer for Attention-based Acoustic-to-word Speech Recognition

Abstract

Acoustic-to-word speech recognition based on attention-based encoder-decoder models achieves better accuracies with much lower latency than the conventional speech recognition systems. However, acoustic-to-word models require a very large amount of training data and it is difficult to prepare one for a new domain such as elderly speech. To address the problem, we propose domain adaptation based on transfer learning with layer freezing. Layer freezing first pre-trains a network with the source domain data, and then a part of parameters is retrained for the target domain while the rest is fixed. In the attentionbased acoustic-to-word model, the encoder part is frozen to maintain the generality, and only the decoder part is retrained to adapt to the target domain. This substantially allows for adaptation of the latent linguistic capability of the decoder to the target domain. Using a large-scale Japanese spontaneous speech corpus as source, the proposed method is applied to three target domains: a call center task and two voice search tasks by adults and by elderly. The models trained with the proposed method achieved better accuracy than the baseline models, which are trained from scratch or entirely retrained with the target domain.

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

“…Most adaptation technologies discussed in this paper can also be applied to domain adaptation [154], [232]- [235]. When the amount of adaptation data is limited, a common practice is adapting only partial layers of the network [236]. To let the adapted model still perform well on the source domain, Moriya et al [237] proposed progressive neural networks by adding an additional model column to the original model for each new domain and only update the new model column with the new domain data.…”

Joint Maximization Decoder with Neural Converters for Fully Neural Network-Based Japanese Speech Recognition

“…In this paper, we use the attention-based encoder-decoder model [23][24][25] for an endto-end ASR system. Our implementation of the model is based on [4,8], and summarized in this section.…”

“…Recent automatic speech recognition (ASR) systems can map acoustic features to word sequences directly; called acoustic-toword (A2W) end-to-end ASR, the approach is based on a fully neural network (FNN) -based architecture [1][2][3][4][5][6][7][8]. Unfortunately, end-to-end ASR systems are not robust to out-of-vocabulary (OOV) words because the number of NN outputs, which correspond to word entries, is fixed.…”

“…Therefore, there are many more characters in Japanese texts than, for example, graphemes in English and OOVs exist at the character-level in external language resources. This means that end-to-end ASR systems need to be re-trained using new data pairs of speech and text every time a new word/character (OOV) is used [7,8]. We want a framework that makes it easy to extend the vocabulary of FNNbased systems by using just text-based language resources.…”

Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End ASR Models

“…However, the pruning algorithm is applied for the entire network, and there has been no investigation into the subnetwork-wise parameter freezing. Some studies on domain adaptation of ASR models have shown that updating only a part of the layers improves the performance on the target domain [16,17]. However, there has been no research on the performance against catastrophic forgetting.…”