Paraphrastic language models

Abstract

In natural languages multiple word sequences can represent the same underlying meaning. Only modelling the observed surface word sequence can result in poor context coverage, for example, when using n-gram language models (LM). To handle this issue, this paper presents a novel form of language model, the paraphrastic LM. A phrase level transduction model that is statistically learned from standard text data is used to generate paraphrase variants. LM probabilities are then estimated by maximizing their marginal probability. Significant error rate reductions of 0.5%-0.6% absolute were obtained on a state-ofthe-art conversational telephone speech recognition task using a paraphrastic multi-level LM modelling both word and phrase sequences.

Paraphrastic neural network language models

“…This advantage can be exploited by many forms of LMs that do not explicitly capture the paraphrastic variability in natural languages. These models include, and are not restricted to, back-off n-gram LMs as investigated in previous research [18,19,20].…”

“…Weighted finite state transducers (WFST) [24] can be can used to efficiently generate paraphrases [18]. For each training data sentence, the paraphrase word lattice T W ′ is generated using a sequence of WFST composition operations, before being projected onto the word sequence level and compressed via the determinization operation.…”

“…As it is impractical to obtain expert semantic labelling at the phrase level, a distributional similarity [8] based statistical paraphrase extraction scheme that operates on standard text data [14,26,1,21] is employed. The n-gram paraphrase induction algorithm proposed in [18] is used. The co-occurrence counts of two phrases of variable lengths, for example, from one word to four words maximum, sharing the same left and right three word contexts, are used to estimate the paraphrase model.…”

“…This intuitively and interpretable discounting method can be exploited by many different forms of LMs that do not explicitly model the expressive richness of natural languages. In previous research, this technique were used to improve the performance of back-off n-gram LMs [18,20,19].…”

“…Along this line, the use of word level synonym features derived from WordNet-type expert resources [10,12,9,5] have been investigated. In order to model the rich paraphrastic relationship between longer span syntactic structures, such as phrases, without manually deriving the associated expert semantic labelling, a novel form of language model, the paraphrastic LM, was previously proposed in [18]. A phrase level generative model statistically learnt from large amounts of standard text data is used to explicitly generate multiple paraphrase variants for each training data sentence.…”