Evaluation of a language model using a clustered model backoff

Abstract

In this paper, we describe and evaluate a language model using word classes automatically generated from a word clustering algorithm. Class based language models have been shown to be effective for rapid adaptation, training on small datasets, and reduced memory usage. In terms of model perplexity, prior work has shown diminished returns for class based language models constructed using very large training sets. This paper describes a method of using a class model as a backoff to a bigram model which produced significant benefits even when trained from a large text corpus. Tests results on the Whisper continuous speech recognition system show that for a given word error rate, the clustered bigram model uses 2/3 fewer parameters compared to a standard bigram model using unigram backoff.

Variable-length categoryn-gram language models

“…Related research, carried out independently and concurrently, has recently been reported in Miller and Alleva (1996). It describes a method that allows backoffs to occur from a word-to a category-based bigram language model.…”

A novel interpolated N-gram language model based on class hierarchy

“…A better approach is to build a language model general enough to better estimate unseen and low frequency events, but specific enough to capture the ambiguous nature of words. A lot of work has been done on this, and the widely used techniques are interpolating the class-based LMs and word-based LMs [4,5,6], and backing-off from word-based LMs to class-based LMs when estimating probabilities of unseen events [7,8]. Performance of the LMs depends on the number of clusters to certain extent.…”