Recurrent Neural Network Language Model Adaptation for Multi-Genre Broadcast Speech Recognition and Alignment

Abstract

Article:Deena, S. orcid.org/0000-0001-5417-0556, Hasan, M., Doulaty, M. et al. (2 more authors) (2019) Recurrent neural network language model adaptation for multi-genre broadcast speech recognition and alignment. IEEE/ACM Transactions on Audio, Speech and Language Processing, 27 (3).Abstract-Recurrent neural network language models (RNNLMs) generally outperform n-gram language models when used in automatic speech recognition. Adapting RNNLMs to new domains is an open problem and current approaches can be categorised as either feature-based or model-based. In feature-based adaptation, the input to the RNNLM is augmented with auxiliary features whilst model-based adaptation includes model fine-tuning and the introduction of adaptation layer(s) in the network. In this paper, the properties of both types of adaptation are investigated on multi-genre broadcast speech recognition. Existing techniques for both types of adaptation are reviewed and the proposed techniques for model-based adaptation, namely the linear hidden network (LHN) adaptation layer and the K-component adaptive RNNLM, are investigated. Moreover, new features derived from the acoustic domain are investigated for RNNLM adaptation. The contributions of this paper include two hybrid adaptation techniques: the fine-tuning of feature-based RNNLMs and a feature-based adaptation layer. Moreover, the semi-supervised adaptation of RNNLMs using genre information is also proposed. The ASR systems were trained using 700h of multi-genre broadcast speech. The gains obtained when using the RNNLM adaptation techniques proposed in this work are consistent when using RNNLMs trained on an in-domain set of 10M words and on a combination of in-domain and out-of-domain sets of 660M words, with approx. 10% perplexity and 2% relative word error rate improvements on a 28.3h. test set. The best RNNLM adaptation techniques for ASR are also evaluated on a lightly supervised alignment of subtitles task for the same data, where the use of RNNLM adaptation leads to an absolute increase in the F-measure of 0.5%.

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

“…Neural network language modelling [270] has become stateof-the-art, in particular recurrent neural network language models (RNNLMs) [271]. There has been a range of work on adaptation of RNNLMs, including the use of topic or genre information as auxiliary features [272], [273] or combined as marginal distributions [274], domain specific embeddings [275], and the use of curriculum learning and fine-tuning to take account of shifting contexts [276], [277]. Approaches based on acoustic model adaptation, such as LHUC [277] and LHN [273], have also been explored.…”

End-to-End Amdo-Tibetan Speech Recognition Based on Knowledge Transfer

“…The decoding process of the LAS model, which is based on output, is different from that of weighted finite-state transducer (WFST), which is based on the frames [31]- [33]. When the Speller starts decoding, the first output token is <sos>, which means the beginning of the sentence.…”

Latent Dirichlet Allocation Based Acoustic Data Selection for Automatic Speech Recognition

“…The idea is that aLDA captures acoustic similarities in the data and tLDA can further help with linguistic content's similarity. tLDA is already shown to improve classification accuracy in LDA based acoustic information retrieval [5] as well as language modelling tasks [15,16,17,18,19]. Training tLDA models followed a similar procedure to aLDA and a comparable number of latent topics and vocabulary size was used.…”