Performance analysis of Neural Networks in combination with n-gram language models

Oparin, Ilya; Sundermeyer, Martin; Ney, Hermann; Gauvain, Jean‐Luc

doi:10.1109/icassp.2012.6289044

Cited by 22 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A back-off to lower order distributions is still required. Techniques that can represent n-gram probabilities in a continuous space, such as NNLMs, can alleviate this problem [23,20]. Hence, in order to to leverage the strengths of both models, the combination between paraphrastic LMs and NNLMs is investigated in this paper.…”

Section: Introductionmentioning

confidence: 99%

Paraphrastic language models and combination with neural network language models

Liu

Gales

Woodland

2013

2013 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

In natural languages multiple word sequences can represent the same underlying meaning. Only modelling the observed surface word sequence can result in poor context coverage, for example, when using n-gram language models (LM). To handle this issue, paraphrastic LMs were proposed in previous research and successfully applied to a US English conversational telephone speech transcription task. In order to exploit the complementary characteristics of paraphrastic LMs and neural network LMs (NNLM), the combination between the two is investigated in this paper. To investigate paraphrastic LMs' generalization ability to other languages, experiments are conducted on a Mandarin Chinese broadcast speech transcription task. Using a paraphrastic multi-level LM modelling both word and phrase sequences, significant error rate reductions of 0.9% absolute (9% relative) and 0.5% absolute (5% relative) were obtained over the baseline n-gram and NNLM systems respectively, after a combination with word and phrase level NNLMs.

show abstract

Section: Introductionmentioning

confidence: 99%

Paraphrastic language models and combination with neural network language models

Liu

Gales

Woodland

2013

2013 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

show abstract

“…A fair comparison is however possible on smaller setups. The reader may find a discussion for which type of events NNLMs do better that -gram LMs and vice versa in [25]. used, all words are clustered).…”

Section: Nnlm Configurationsmentioning

confidence: 99%

Structured Output Layer Neural Network Language Models for Speech Recognition

Oparin

Allauzen

et al. 2013

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

This paper extends a novel neural network language model (NNLM) which relies on word clustering to structure the output vocabulary: Structured OUtput Layer (SOUL) NNLM. This model is able to handle arbitrarily-sized vocabularies, hence dispensing with the need for shortlists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which is based on the continuous word representation determined by the NNLM. Mandarin and Arabic data are used to evaluate the SOUL NNLM accuracy via speech-to-text experiments. Well tuned speech-to-text systems (with error rates around 10%) serve as the baselines. The SOUL model achieves consistent improvements over a classical shortlist NNLM both in terms of perplexity and recognition accuracy for these two languages that are quite different in terms of their internal structure and recognition vocabulary size. An enhanced training scheme is proposed that allows more data to be used at each training iteration of the neural network.Index Terms-Automatic speech recognition, neural network language model, speech-to-text.

show abstract

“…n-gram LMs have been the dominant language models during the last several decades [25], [26]. Uni-RNNLMs were shown to present different and complementary modeling ability to n-gram LMs [27], [28]. Improved performance can be obtained by interpolating n-gram and uni-RNNLMs [27].…”

Section: A Language Model Interpolationmentioning

confidence: 99%

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition

Chen

Liu

Wang

et al. 2019

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Language modelling is a crucial component in a wide range of applications including speech recognition. Language models (LMs) are usually constructed by splitting a sentence into words and computing the probability of a word based on its word history. This sentence probability calculation, making use of conditional probability distributions, assumes that there is little impact from approximations used in the LMs including: the word history representations; and approaches to handle finite training data. This motivates examining models that make use of additional information from the sentence. In this work future word information, in addition to the history, is used to predict the probability of the current word. For recurrent neural network LMs (RNNLMs) this information can be encapsulated in a bi-directional model. However, if used directly this form of model is computationally expensive when training on large quantities of data, and can be problematic when used with word lattices. This paper proposes a novel neural network language model structure, the succeeding-word RNNLM, su-RNNLM, to address these issues. Instead of using a recurrent unit to capture the complete future word contexts, a feed-forward unit is used to model a fixed finite number of succeeding words. This is more efficient in training than bi-directional models and can be applied to lattice rescoring. The generated lattices can be used for downstream applications, such as confusion network decoding and keyword search. Experimental results on speech recognition and keyword spotting tasks illustrate the empirical usefulness of future word information, and the flexibility of the proposed model to represent this information.

show abstract

Performance analysis of Neural Networks in combination with n-gram language models

Cited by 22 publications

References 14 publications

Paraphrastic language models and combination with neural network language models

Paraphrastic language models and combination with neural network language models

Structured Output Layer Neural Network Language Models for Speech Recognition

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition

Contact Info

Product

Resources

About