Cluster-based language models for distributed retrieval

Abstract

E ective retrieval in a distributed environment is an important but di cult problem. Lack of e ectiveness appears to have three causes. First, collection selection based on word histograms is not appropriate for heterogeneous collections. Second, relevant documents are scattered over many collections and searching a few collections misses many relevant documents. Third, most existing collection selection metrics lack sound theoretical justi cations and hence may not bewell tuned to the problem. We propose a new approach to distributed retrieval based on document clustering and language modeling. Document clustering is used to organize collections around topics. Language modeling is used to properly represent topics and e ectively select the right topics for a query. Based on these ideas, three methods are proposed to suit di erent e n vironments. We show that all three methods improve e ectiveness of distributed retrieval.

The impact of database selection on distributed searching

“…Viles and French [9,19] showed that dissemination of collection information increased retrieval effectiveness. Xu and Croft [23] explored cluster-based language models, investigating different ways to construct database selection indexes.…”

“…However, in recent work, Xu and Croft [23] discuss the possibility that retrieval performance in a distributed environment may exceed performance in a centralized environment. In that work, Xu and Croft were pessimistic about the potential to achieve both retrieval efficiency and effectiveness in heterogeneous distributed environments.…”

Search for expertise : going beyond direct evidence

“…While methods just merging collection documents to get aggregated statistics are among earliest known (Zobel, 1997;Xu and Croft, 1999), the most popular approach uses a task-specific tfidf approach. Document frequency in the collection is used instead of the sum of term frequencies, and inverted document frequency is approximated by inverted collection frequency (Callan et al, 1995).…”

Investigation of the Effectiveness of Cross-Media Indexing

“…When applied to the problem of measuring the distance between two term distributions as in Language Modeling [23], KL estimates the relative entropy between the probability of a term t occurring in the actual collection Θ c (i.e. p(t|Θ c )), and the probability of the term t occurring in the estimated Topic Language Model Θ d (i.e.…”

“…A nonzero constant α is introduced to alleviate the zero probability [23]. The smaller the KL divergence the closer the document is to the actual collection.…”