Latent Semantic Indexing: A Probabilistic Analysis

Abstract

Latent semantic indexing (LSI) is an information retrieval technique based on the spectral analysis of the term-document matrix, whose empirical success had heretofore been without rigorous prediction and explanation. We prove that, under certain conditions, LSI does succeed in capturing the underlying semantics of the corpus and achieves improved retrieval performance. We propose the technique of random projection as a way of speeding up LSI. We complement our theorems with encouraging experimental results. We also argue that our results may be viewed in a more general framework, as a theoretical basis for the use of spectral methods in a wider class of applications such as collaborative filtering. Academic Press

The stochastic approach for link-structure analysis (SALSA) and the TKC effect

“…Term-based search engines face both classical problems in information retrieval, as well as problems specific to the World Wide Web setting, when handling broad-topic queries. The classic problems include the following issues [4,20]. ž Synonymy -retrieving documents containing the term 'car' when given the query 'automobile'.…”

On Approximation Algorithms for Data Mining Applications

“…In [88], a model for documents is developed on which the LSI method is evaluated. In this model, each document is built out of a number of different topics (hidden from the retrieval algorithm).…”

Improving Random Projections Using Marginal Information

“…Because the probability of multiple roots is so small when k ≥ 8 while in the large-scale applications we expect k 10, we suggest not to worry about multiple roots. Also, we will only use the O(k −1 ) term of Var(â MLE ), i.e, (7). Figure 3 presents some simulation results, using two words "THIS" and "HAVE," from some MSN Web crawl data.…”

“…Random projections [1] have been used in machine learning [2][3][4][5][6] and many other applications in data mining and information retrieval, e.g., [7][8][9][10][11][12].…”