Class phrase models for language modeling

Abstract

Previous attempts to a utomatically determine m ulti-words as the basic unit for language modeling h ave been successful for extending bigram models 10, 9, 2, 8 to improve t he perplexity o f t he language model and or the w ord accuracy of the speech d ecoder. However, none o f t hese techniques gave improvements over the trigram model so far, except for the rather controlled ATIS task 8 . We t herefore propose an algorithm, that minimizes the perplexity improvement o f a bigram model directly. The n ew algorithm is able to reduce the trigram perplexity a n d also achieves word accuracy improvements in the V erbmobil task. It is the n atural counterpart of successful word classi cation algorithms for language modeling 4, 7 that minimize the leaving-one-out bigram perplexity. W e also give some d etails on the usage of class nding t echniques and m-gram models, which can be crucial to s u ccessful applications of this technique.

Maximizing Component Quality in Bilingual Word-Aligned Segmentations

“…By using (6) as the objective function, we observed that the resulting segmentations yield promising applications in n-gram topic modeling, named entity recognition and Chinese segmentation. However, in the spirit of Ries et al (1996), attempts to minimize perplexity instead of maximizing (6), resulted in larger segments and the segment quality definition of Section 1 was not met.…”

Melodic analysis with segment classes

“…In natural language, statistical modeling has mainly been applied to modeling word sequences (Jurafsky & Martin, 2000). The idea of using class models for phrases rather than words was presented by Ries, Buø and Waibel (1996), with the result called class phrase models. There, words were grouped into short phrases, and the phrase class is the sequence of word classes in the phrase.…”

“…The analogy to the work described here is revealed by considering words as notes and phrases as segments. The phrase representation used by Ries, Buø and Waibel (1996) can be viewed as an instance of the lift viewpoint constructor, where the basic event attributes rather than abstract event classes are lifted. Therefore the viewpoint representation may prove useful in natural language modeling.…”

Statistical language modeling with a class-basedn-multigram model

“…In grambased approaches (Riccardi, Pieraccini & Bocchieri, 1996;Hu, Turin & Brown, 1997;Ristad & Thomas, 1997;Siu & Ostendorf, 1997;Niesler & Woodland, 1999), models take into account variable-length dependencies by conditioning the probability of each word with a context of variable length. In contrast, in phrase-based approaches (Suhm & Waibel, 1994;Deligne & Bimbot, 1995;Masataki & Sagisaka, 1996;Ries, Buo & Waibel, 1996;Matsunaga & Sagayama, 1997;Riccardi & Bangalore, 1998;Siu, 1998), sentences are structured into variable-length phrases and probabilities are assigned to phrases instead of words. The probability of each phrase may be conditioned by the preceding phrases, in just the same way that the probability of a word is conditioned by the preceding words in a gram-based framework.…”