Evaluating hierarchical hybrid statistical language models

Abstract

We introduce in this paper a hierarchical hybrid statistical language model, represented as a collection of local models plus a general model that binds together the local ones. The model provides a unified framework for modelling language both above and below the word level, and we exemplify with models of both kinds for a large vocabulary task domain. To our knowledge this is the first paper to report an extensive evaluation of the improvements achieved from the use of local models within a hierarchical framework in comparison with a conventional word-based trigram model.

Hierarchical statistical language models: experiments on in-domain adaptation

“…We compute the APP value on the full test set, and thus we can compare two models with different vocabularies. We again refer the reader to [5] for more details on our evaluation procedure.…”

“…Note that the probability depends only on the fact that the word seen in the test data is a number, and not on which number it is; in particular, numbers not encountered in the training data will receive the same probability as the ones encountered. For a comparison between this probability model and a more conventional one, see [5].…”