Stochastic Analysis of Lexical and Semantic Enhanced Structural Language Model

Abstract

Abstract. In this paper, we present a directed Markov random field model that integrates trigram models, structural language models (SLM) and probabilistic latent semantic analysis (PLSA) for the purpose of statistical language modeling. The SLM is essentially a generalization of shift-reduce probabilistic push-down automata thus more complex and powerful than probabilistic context free grammars (PCFGs). The added context-sensitiveness due to trigrams and PLSAs and violation of tree structure in the topology of the underlying random field model make the inference and parameter estimation problems plausibly intractable, however the analysis of the behavior of the lexical and semantic enhanced structural language model leads to a generalized inside-outside algorithm and thus to rigorous exact EM type re-estimation of the composite language model parameters.

A Scalable Distributed Syntactic, Semantic, and Lexical Language Model

“…When combining n-gram, m-SLM, and PLSA together to build a composite generative language model under the directed MRF paradigm (Wang et al 2005b(Wang et al , 2006, the composite language model is simply a complicated generative model that has four operators: WORD-PREDICTOR, TAGGER, CONSTRUCTOR, and SEMANTIZER. The TAGGER and CONSTRUCTOR in SLM and the SEMANTIZER in PLSA remain unchanged; the WORD-PREDICTORs in n-gram, m-SLM, and PLSA, however, are combined to form a stronger WORD-PREDICTOR that generates the next word, w k+1 , not only depending on the m most recently exposed headwords h −1 −m in the word-parse k-prefix but also its n-gram history w k k−n+2 and its semantic content g k+1 .…”

“…The composite n-gram/m-SLM/PLSA language model can be formulated as a rather complex chain-tree-table directed MRF model (Wang et al 2006) A composite n-gram/m-SLM/PLSA language model where the hidden information is the parse tree T and semantic content g. The n-gram encodes local word interactions, the m-SLM models the sentence's syntactic structure, and the PLSA captures the document's semantic content; all interact together to constrain the generation of natural language. The WORD-PREDICTOR generates the next word w k+1 with probability p…”

“…When applying this to the Wall Street Journal corpus with 40 million tokens, they achieved moderate perplexity reduction. Because the probabilistic dependency structure in a structured language model (SLM) (Chelba 2000;Chelba and Jelinek 2000) is more complex and powerful than that in a PCFG, Wang et al (2006) studied the stochastic properties for the composite language model that integrates n-gram, SLM, and PLSA under the directed MRF framework (Wang et al 2005b) and derived another generalized inside-outside algorithm to train a composite ngram, SLM, and PLSA language model from a general expectation maximization (EM) (Dempster, Laird, and Rubin 1977) algorithm by following Jelinek's ingenious definition of the inside and outside probabilities for SLM (Jelinek 2004). Again, the generalized inside-outside algorithm alters Jelinek's inside-outside algorithm with modular modification and has the same sixth order of sentence-length time complexity.…”

“…In this article, we study the same composite n-gram, SLM, and PLSA model under the directed MRF framework as in Wang et al (2006). The composite n-gram/ SLM/PLSA language model under the directed MRF paradigm is first introduced in Section 2.…”

“…The composite n-gram/ SLM/PLSA language model under the directed MRF paradigm is first introduced in Section 2. In Section 3, instead of using the sixth order generalized inside-outside algorithm proposed in Wang et al (2006), we show how to train this composite model via an N-best list approximate EM algorithm that has linear time complexity and a follow-up EM algorithm to improve word prediction power. We prove the convergence of the N-best list approximate EM algorithm.…”