Structure and Performance of a Dependency Language Model

Chelba, Ciprian; Engle, David M.; Jelinek, F.; Jiménez, Víctor M.; Khudanpur, Sanjeev; Mangu, Lidia; Printz, Harry; Ristad, Eric Sven; Rosenfeld, Ronald; Stolcke, Andreas; Wu, Dekai

doi:10.21236/ada640606

Cited by 50 publications

(53 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…How these constraints are incorporated varies from estimating -gram probabilities from grammar-generated data [70] to computing a linear interpolation of the two models [43]. Most recently, syntactic information has been used specifically to determine equivalence classes on the -gram history, resulting in so-called dependency language models [19], [56], sometimes also referred to as structured language models [20], [42], [66].…”

Section: B Syntactically Driven Span Extensionmentioning

confidence: 99%

Exploiting latent semantic information in statistical language modeling

2000

View full text Add to dashboard Cite

Section: B Syntactically Driven Span Extensionmentioning

confidence: 99%

Exploiting latent semantic information in statistical language modeling

2000

View full text Add to dashboard Cite

“…As shown in Table 4, although the size of CM (i.e., # of dependencies) is much larger, the improvement is very limited. On the other end of the spectrum, we have models that use sophisticated syntactic structure, such as dependencybased models [4,5] and constituency-based models [2,3]. They all use syntactic grammars for parsing and the parsing model is estimated from a manually annotated training data (i.e.…”

Section: Discussion On Term Dependencies With or Without Linguistic mentioning

confidence: 99%

“…[4]), we assume that the sum ∑ L P(Q, L|D) over all the possible Ls is dominated by a single term L * which is the most probable linkage of the query Q. Below we simply use L to represent L * .…”

Section: A New Modelmentioning

confidence: 99%

Dependence language model for information retrieval

Gao

Nie

et al. 2004

Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

186

123

View full text Add to dashboard Cite

This paper presents a new dependence language modeling approach to information retrieval. The approach extends the basic language modeling approach based on unigram by relaxing the independence assumption. We integrate the linkage of a query as a hidden variable, which expresses the term dependencies within the query as an acyclic, planar, undirected graph. We then assume that a query is generated from a document in two stages: the linkage is generated first, and then each term is generated in turn depending on other related terms according to the linkage. We also present a smoothing method for model parameter estimation and an approach to learning the linkage of a sentence in an unsupervised manner. The new approach is compared to the classical probabilistic retrieval model and the previously proposed language models with and without taking into account term dependencies. Results show that our model achieves substantial and significant improvements on TREC collections.

show abstract

“…Jurafsky et al [71] use stochastic CFGs (SCFGs) to extend the corpus for training and interpolates SCFG probabilities with bi-gram probabilities. Chelba et al [24] use a dependency grammar framework with maximum entropy models to constrain word prediction by the linguistically related words in the past. The most important instance of LMs that use syntactic structure is presented in [25].…”

Section: Language Modelmentioning

confidence: 99%

“…The first attempts are based on using context free grammars (CFGs) [27,137,71]. The main contribution of structured LMs is started with Chelba et al [24] in which a dependency grammar framework with maximum entropy models is used to constrain the word prediction by the linguistically related words in the past.…”

Section: Structured Lmsmentioning

confidence: 99%

Semantic language models with deep neural networks

Bayer

Riccardi

2016

Computer Speech & Language

View full text Add to dashboard Cite

Spoken language systems (SLS) communicate with users in natural language through speech. There are two main problems related to processing the spoken input in SLS. The first one is automatic speech recognition (ASR) which recognizes what the user says. The second one is spoken language understanding (SLU) which understands what the user means. We focus on the language model (LM) component of SLS. LMs constrain the search space that is used in the search for the best hypothesis. Therefore, they play a crucial role in the performance of SLS.It has long been discussed that an improvement in the recognition performance does not necessarily yield a better understanding performance. Therefore, optimization of LMs for the understanding performance is crucial. In addition, long-range dependencies in languages are hard to handle with statistical language models. These two problems are addressed in this thesis.We investigate two different LM structures. The first LM that we investigate enable SLS to understand better what they recognize by searching the ASR hypotheses for the best understanding performance. We refer to these models as joint LMs. They use lexical and semantic units jointly in the LM. The second LM structure uses the semantic context of an utterance, which can also be described as "what the system understands", to search for a better hypothesis that improves the recognition and the understanding performance. We refer to these models as semantic LMs (SELMs). SELMs use features that are based on a well established theory of lexical semantics, namely the theory of frame semantics. They incorporate the semantic features which are extracted from the ASR hypothesis into the LM and handle long-range dependencies by using the semantic relationships between words and semantic context. ASR noise is propagated to the semantic features, to suppress this noise we introduce the use of deep semantic encodings for semantic feature extraction. In this way, SELMs optimize both the recognition and the understanding performance.

show abstract

Structure and Performance of a Dependency Language Model

Cited by 50 publications

References 6 publications

Exploiting latent semantic information in statistical language modeling

Exploiting latent semantic information in statistical language modeling

Dependence language model for information retrieval

Semantic language models with deep neural networks

Contact Info

Product

Resources

About