Exploiting latent semantic information in statistical language modeling

Bellegarda, Jerome R.

doi:10.1109/5.880084

Cited by 268 publications

(170 citation statements)

References 48 publications

Supporting

Mentioning

165

Contrasting

Unclassified

Order By: Relevance

“…Note that the benefit of the maximum entropy combination method is that the cluster nodes behave like latent variables in a mixture model for "soft clustering", instead of the "hard clusters" created by methods like K-means used in Bellegarda (2000). Figure 2 shows an extended semantic smoothed version of the model that incorporates additional word cluster variables within each topic, and additional topic cluster variables with each document.…”

Section: Semantic Smoothingmentioning

confidence: 99%

“…This approach has several advantages over other methods for statistical modeling, such as introducing less data fragmentation (as in decision tree learning), requiring fewer independence assumptions (as in naive Bayes models), and exploiting a principled technique for automatic feature weighting. The major weakness with maximum entropy methods, however, are that they can only model distributions over explicitly observed features, whereas in natural language we encounter hidden semantic (Bellegarda, 2000;Hofmann, 2001) and syntactic information (Chelba & Jelinek, 2000) which we do not observe directly.…”

Section: Introductionmentioning

confidence: 99%

“…In language modeling research, there exists a body of ad hoc tricks which are known to make significant improvements. For example, (Bellegarda, 2000) proposes an ad hoc combination of tri-gram and latent semantic analysis (LSA) models, by using the perplexity formula…”

Section: Extensionsmentioning

confidence: 99%

“…For example, the structural language model (Chelba & Jelinek, 2000;Roark, 2001) effectively exploits relevant syntactic regularities to improve the perplexity score of N-gram models, and the semantic language model (Bellegarda, 2000;Bengio et al, 2003;Hofmann, 2001) exploits document-level semantic regularities to acheive similar improvements. Although each of these language models outperforms simple N-grams, they each only capture specific linguistic phenomena.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Combining Statistical Language Models via the Latent Maximum Entropy Principle

Wang

Schuurmans

Peng³

et al. 2005

Mach Learn

View full text Add to dashboard Cite

Abstract. We present a unified probabilistic framework for statistical language modeling which can simultaneously incorporate various aspects of natural language, such as local word interaction, syntactic structure and semantic document information. Our approach is based on a recent statistical inference principle we have proposed-the latent maximum entropy principle-which allows relationships over hidden features to be effectively captured in a unified model. Our work extends previous research on maximum entropy methods for language modeling, which only allow observed features to be modeled. The ability to conveniently incorporate hidden variables allows us to extend the expressiveness of language models while alleviating the necessity of pre-processing the data to obtain explicitly observed features. We describe efficient algorithms for marginalization, inference and normalization in our extended models. We then use these techniques to combine two standard forms of language models: local lexical models (Markov N-gram models) and global document-level semantic models (probabilistic latent semantic analysis). Our experimental results on the Wall Street Journal corpus show that we obtain a 18.5% reduction in perplexity compared to the baseline tri-gram model with Good-Turing smoothing.

show abstract

Section: Semantic Smoothingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Extensionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Combining Statistical Language Models via the Latent Maximum Entropy Principle

Wang

Schuurmans

Peng³

et al. 2005

Mach Learn

View full text Add to dashboard Cite

show abstract

“…[1]. It can be shown that NMF provides a clear interpretation of concepts such as abstraction in terms of linear algebraic operations [23].…”

Section: Brief Description Of the Modelmentioning

confidence: 99%

On a Computational Model for Language Acquisition: Modeling Cross-Speaker Generalisation

Bosch

Driesen

hamme

et al. 2009

Text, Speech and Dialogue

View full text Add to dashboard Cite

Abstract. The discovery of words by young infants involves two interrelated processes: (a) the detection of recurrent word-like acoustic patterns in the speech signal, and (b) cross-modal association between auditory and visual information. This paper describes experimental results obtained by a computational model that simulates these two processes. The model is able to build word-like representations on the basis of multimodal input data (stimuli) without the help of an a priori specified lexicon. Each input stimulus consists of a speech signal accompanied by an abstract visual representation of the concepts referred to in the speech signal. In this paper we investigate how internal representations generalize across speakers. In doing so, we also analyze the cognitive plausibility of the model.

show abstract

Task Estimation Using Latent Semantic Analysis of Visual Scenes and Spoken Words

Kimura

Sawada

Iribe

et al. 2014

Elect Comm in Japan

View full text Add to dashboard Cite

In this paper, we propose a task estimation method based on multiple subspaces extracted from multimodal information of image objects in visual scenes and spoken words in dialogue appearing in the same task. The multiple subspaces are obtained by using latent semantic analysis (LSA). In the proposed method, a task vector composed of spoken words and the frequencies of image-object appearances are extracted first, and then similarities among the input task vector and reference subspaces of different tasks are compared. Experiments are conducted on the identification of game tasks. The experimental results show that the proposed method with multimodal information outperforms the method in which only the single modality of image or spoken dialogue is applied. The proposed method achieves accurate performance even if less spoken dialogue is applied. C⃝ 2014 Wiley Periodicals, Inc. Electron Comm Jpn, 97(6): 33-42, 2014; Published online in Wiley Online Library (wileyonlinelibrary.com).

show abstract

Exploiting latent semantic information in statistical language modeling

Cited by 268 publications

References 48 publications

Combining Statistical Language Models via the Latent Maximum Entropy Principle

Combining Statistical Language Models via the Latent Maximum Entropy Principle

On a Computational Model for Language Acquisition: Modeling Cross-Speaker Generalisation

Task Estimation Using Latent Semantic Analysis of Visual Scenes and Spoken Words

Contact Info

Product

Resources

About