Efficient lattice rescoring using recurrent neural network language models

Liu, Xunying; Wang, Yongqiang; Xie, Changsheng; Gales, Mark J. F.; Woodland, Philip C.

doi:10.1109/icassp.2014.6854535

Cited by 74 publications

(67 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hybrid input representation can considerably save both training and decoding time while still achieving slightly better recognition accuracy. In the future, we plan to apply more complex lattice re-scoring algorithms, such as the one described in [20], to the hybrid RNNLM to further improve recognition performance. A cache-based RNNLM which can be used straightaway in the first-pass decoding [21] will also be considered.…”

Section: Discussionmentioning

confidence: 99%

A hybrid input-type recurrent neural network for LVCSR language modeling

Chunwijitra

Chotimongkol

Wutiwiwatchai

2016

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Substantial amounts of resources are usually required to robustly develop a language model for an open vocabulary speech recognition system as out-of-vocabulary (OOV) words can hurt recognition accuracy. In this work, we applied a hybrid lexicon of word and sub-word units to resolve the problem of OOV words in a resource-efficient way. As sub-lexical units can be combined to form new words, a compact set of hybrid vocabulary can be used while still maintaining a low OOV rate. For Thai, a syllable-based unit called pseudo-morpheme (PM) was chosen as a sub-word unit. To also benefit from different levels of linguistic information embedded in different input types, a hybrid recurrent neural network language model (RNNLM) framework is proposed. An RNNLM can model not only information from multiple-type input units through a hybrid input vector of words and PMs, but can also capture long context history through recurrent connections. Several hybrid input representations were also explored to optimize both recognition accuracy and computational time. The hybrid LM has shown to be both resource-efficient and well-performed on two Thai LVCSR tasks: broadcast news transcription and speech-to-speech translation. The proposed hybrid lexicon can constitute an open vocabulary for Thai LVCSR as it can greatly reduce the OOV rate to less than 1 % while using only 42 % of the vocabulary size of the word-based lexicon. In terms of recognition performance, the best proposed hybrid RNNLM, which uses a mixed word-PM input, obtained 1.54 % relative WER reduction when compared with a conventional word-based RNNLM. In terms of computational time, the best hybrid RNNLM has the lowest training and decoding time among all RNNLMs including the word-based RNNLM. The overall relative reduction on WER of the proposed hybrid RNNLM over a traditional n-gram model is 6.91 %.

show abstract

Section: Discussionmentioning

confidence: 99%

A hybrid input-type recurrent neural network for LVCSR language modeling

Chunwijitra

Chotimongkol

Wutiwiwatchai

2016

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

show abstract

“…As RNNLMs use a vector representation of full histories, they are mostly used for N-best list rescoring. For more efficient lattice rescoring using RNNLMs, approximation schemes, for example, based on clustering among complete histories [14] can be used.…”

Section: Recurrent Neural Network Lmsmentioning

confidence: 99%

Improving the training and evaluation efficiency of recurrent neural network language models

Chen

Liu

Gales

et al. 2015

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Recurrent neural network language models (RNNLMs) are becoming increasingly popular for speech recognition. Previously, we have shown that RNNLMs with a full (non-classed) output layer (F-RNNLMs) can be trained efficiently using a GPU giving a large reduction in training time over conventional class-based models (C-RNNLMs) on a standard CPU. However, since test-time RNNLM evaluation is often performed entirely on a CPU, standard F-RNNLMs are inefficient since the entire output layer needs to be calculated for normalisation. In this paper, it is demonstrated that C-RNNLMs can be efficiently trained on a GPU, using our spliced sentence bunch technique which allows good CPU test-time performance (42× speedup over F-RNNLM). Furthermore, the performance of different classing approaches is investigated. We also examine the use of variance regularisation of the softmax denominator for F-RNNLMs and show that it allows F-RNNLMs to be efficiently used in test (56× speedup on a CPU). Finally the use of two GPUs for F-RNNLM training using pipelining is described and shown to give a reduction in training time over a single GPU by a factor of 1.6×.

show abstract

“…In recent years, the use of RNNLM have shown significant improvements over the traditional n-gram models (Sundermeyer et al, 2013). (Mikolov et al, 2010) and (Liu et al, 2014) have shown significant improvements in speech recognition accuracy using RNNLM . Shi (2012) also showed the benefits of using RNNLM with contextual and linguistic features.…”

Section: Related Workmentioning

confidence: 99%

“…We have integrated the approach of re-scoring n-best output using RNNLM which has also been shown to be helpful by (Liu et al, 2014). Shi (2012) also showed the benefits of using RNNLM with contextual and linguistic features.…”

Section: Re-ranking Experimentsmentioning

confidence: 99%

Reducing the Impact of Data Sparsity in Statistical Machine Translation

Singla

Sachdeva

Bangalore

et al. 2014

Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation

View full text Add to dashboard Cite

Morphologically rich languages generally require large amounts of parallel data to adequately estimate parameters in a statistical Machine Translation(SMT) system. However, it is time consuming and expensive to create large collections of parallel data. In this paper, we explore two strategies for circumventing sparsity caused by lack of large parallel corpora. First, we explore the use of distributed representations in an Recurrent Neural Network based language model with different morphological features and second, we explore the use of lexical resources such as WordNet to overcome sparsity of content words.

show abstract

Efficient lattice rescoring using recurrent neural network language models

Cited by 74 publications

References 19 publications

A hybrid input-type recurrent neural network for LVCSR language modeling

A hybrid input-type recurrent neural network for LVCSR language modeling

Improving the training and evaluation efficiency of recurrent neural network language models

Reducing the Impact of Data Sparsity in Statistical Machine Translation

Contact Info

Product

Resources

About