Generalized algorithms for constructing statistical language models

Abstract

Recent text and speech processing applications such as speech mining raise new and more general problems related to the construction of language models. We present and describe in detail several new and efficient algorithms to address these more general problems and report experimental results demonstrating their usefulness. We give an algorithm for computing efficiently the expected counts of any sequence in a word lattice output by a speech recognizer or any arbitrary weighted automaton; describe a new technique for creating exact representations of ¢-gram language models by weighted automata whose size is practical for offline use even for a vocabulary size of about 500,000 words and an ¢-gram order ¢ ¤ £ ¦ ¥ ; and present a simple and more general technique for constructing class-based language models that allows each class to represent an arbitrary weighted automaton. An efficient implementation of our algorithms and techniques has been incorporated in a general software library for language modeling, the GRM Library, that includes many other text and grammar processing functionalities.

Discriminative segmental cascades for feature-rich phone recognition

“…Continuing the above example, we can σ-compose the decoding graph H with a bigram language model (LM) L in its FST form [15] with a slight modification. We require the output labels of the LM FST to include the history labels alongside the current label.…”

Learning the Structure of Variable-Order CRFs: a finite-state perspective

“…Our procedure for building A[W] is sketched in Algorithm 1, where TrieInsert inserts a string in a trie, Pref(W) computes the set of prefixes of the strings in W, 3 LgSuff(v, U) returns the longest suffix of v in U, and FailureTrans is a special ε-transition used only when no labelled transition exists (Allauzen et al, 2003). 4 Each state (or pattern prefix) v in A[W] is associated with a set of feature functions {f u,g , ∀u ∈ Suff(v), g}.…”

Weighted Automata Algorithms

“…These algorithms are useful in a variety of applications including statistical language modeling [7], parsing [45,44,8], phonological rule compilation [29,30,49], speech recognition [38,43,48], speech synthesis [56,6], image processing [2], bioinformatics [21,9], sequence modeling and prediction [18], optical character recognition [14], and more generally any problem related to sequences and probabilistic models defined over sequences [33,43]. An efficient implementation of these algorithms and several others, including an on-demand implementation when possible, is available from the FSM library (executables only) [47] and the OpenFst library (source and executables) [10].…”