Paraphrastic language models and combination with neural network language models

Abstract

In natural languages multiple word sequences can represent the same underlying meaning. Only modelling the observed surface word sequence can result in poor context coverage, for example, when using n-gram language models (LM). To handle this issue, paraphrastic LMs were proposed in previous research and successfully applied to a US English conversational telephone speech transcription task. In order to exploit the complementary characteristics of paraphrastic LMs and neural network LMs (NNLM), the combination between the two is investigated in this paper. To investigate paraphrastic LMs' generalization ability to other languages, experiments are conducted on a Mandarin Chinese broadcast speech transcription task. Using a paraphrastic multi-level LM modelling both word and phrase sequences, significant error rate reductions of 0.9% absolute (9% relative) and 0.5% absolute (5% relative) were obtained over the baseline n-gram and NNLM systems respectively, after a combination with word and phrase level NNLMs.

Paraphrastic neural network language models

“…Word error rate reductions of 1.3% (8% relative) absolute were obtained on a state-of-the-art large vocabulary speech recognition task. Consistent with the performance improvements previously obtained on back-off n-gram LMs [18,20,19], experimental results presented in this paper suggest the proposed method is also effective in improving the generalization performance of feedforward NNLMs. In contrast, previous research on NNLMs used no explicit paraphrastic modelling [2,27,25,13,22].…”

“…This advantage can be exploited by many forms of LMs that do not explicitly capture the paraphrastic variability in natural languages. These models include, and are not restricted to, back-off n-gram LMs as investigated in previous research [18,19,20].…”

“…This intuitively and interpretable discounting method can be exploited by many different forms of LMs that do not explicitly model the expressive richness of natural languages. In previous research, this technique were used to improve the performance of back-off n-gram LMs [18,20,19].…”

Cross-domain paraphrasing for improving language modelling using out-of-domain data

“…The particular type of LMs considered in this paper can flexibly model paraphrase mapping at the word, phrase and sentence level. As LM probabilities are estimated in the paraphrased domain, they are referred to as paraphrastic language models (PLM) [16,17]. For a L word long word sequence W =< w1, w2, ..., wi, ..., wL > in the training data, rather than maximizing the surface word sequence's log-probability ln P (W) as for conventional LMs, the marginal probability over all paraphrase variant sequences is maximized,…”

“…The statistics required for paraphrastic LM estimation are then accumulated from the paraphrase lattices via a forwardbackward pass. In order to improve phrase coverage, expert semantic labelling provided by resources, such as WordNet [5], can also be used to generate paraphrases [16,17].…”