Just-in-time language modelling

Abstract

Traditional approaches to language modelling have relied on a fixed corpus of text to inform the parameters of a probability distribution over word sequences. Increasing the corpus size often leads to better-performing language models, but no matter how large, the corpus is a static entity, unable to reflect information about events which postdate it. In these pages we introduce an online paradigm which interleaves the estimation and application of a language model. We present a Bayesian approach to online language modelling, in which the marginal probabilities of a static trigram model are dynamically updated to match the topic being dictated to the system. We also describe the architecture of a prototype we have implemented which uses the World Wide Web (WWW) as a source of information, and provide results from some initial proof of concept experiments.

A Gaussian Prior for Smoothing Maximum Entropy Models

“…In this work, we showed that a Gaussian prior can be used to smooth maximum entropy n-gram models to achieve performance equal to or superior to that of all other techniques for 6 The optimal variances Na 2 m for the Gaussian prior found by the Powell search were mostly in the range 1.5 < No 2 m < 5, where N is the size of the training set. Multiplying by N converts the variances from probability space to count space, and discounts are relatively constant in count space over different training set sizes.…”

“…In the original implementation ME-gauss, a different a m is used for each level of the n-gram model. 6 We also considered using a single a over the whole model (ME-gauss-1), and using three parameters <7 TO ,i, o m^i and <7 mj 3+ for each level of the n-gram model, to be applied to m-grams with 1, 2, or 3 or more counts in the training data, respectively. This latter parameterization (ME-gauss-3n) is analogous to the parameterization of modified KneserNey smoothing.…”

“…First, they point out that the modified lower-order distributions of Kneser-Ney smoothing is the primary reason for its superiority among conventional n-gram smoothing algorithms. Recall that these distributions are chosen to satisfy marginal constraints as given in equation (6). However, this set of constraints is identical to the corresponding maximum entropy constraints for (n -1)-grams, as given by equation (9).…”

“…This technique has been applied previously in [6]. Recall that a maximum entropy model is the maximum likelihood model among the set of models given by equation (3).…”

“…For many problems, this type of modeling can be viewed as maximum likelihood (ML) training for exponential models, and like other maximum likelihood methods is prone to overfitting of training data. While several smoothing methods for maximum entropy models have been proposed to address this problem [6,1,7,5], previous results do not make it clear how these smoothing methods compare with smoothing methods for other types of related models.…”