A study of the integration of passage-, document-, and cluster-based information for re-ranking search results

Abstract

Cluster-based and passage-based document retrieval paradigms were shown to be effective. While the former are based on utilizing query-related corpus context manifested in clusters of similar documents, the latter address the fact that a document can be relevant even if only a very small part of it contains query-pertaining information. Hence, cluster-based approaches could be viewed as based on ''expanding'' the document representation, while passage-based approaches can be thought of as utilizing a ''contracted'' document representation. We present a study of the relative benefits of using each of these two approaches, and of the potential merits of their integration. To that end, we devise two methods that integrate whole-document-based, cluster-based and passage-based information. The methods are applied for the re-ranking task, that is, re-ordering documents in an initially retrieved list so as to improve precision at the very top ranks. Extensive empirical evaluation attests to the potential merits of integrating these information types. Specifically, the resultant performance substantially transcends that of the initial ranking; and, is often better than that of a state-of-the-art pseudo-feedback-based query expansion approach.

Rhetorical relations for information retrieval

“…We evaluate our model on the task of re-ranking an initial list of documents, which has been retrieved in response to a query. Re-ranking is a well-known IR practice that can enhance retrieval performance notably [19]. The baseline of our experiments consists of the top 1000 documents retrieved for each query using a state-of-the-art retrieval model (language model with Dirichlet smoothing 1 [9]).…”

Scalable Text Mining with Sparse Generative Models

“…A variety of hierarchically smoothed sequence models are used with text, such as interpolated n-gram models. The smoothing hierarchy can come from sources such as word clusters [Zitouni and Zhou, 2008], the local word context [Chen and Goodman, 1999], or collection structure [McCallum and Nigam, 1999, Zhang et al, 2002, Krikon and Kurland, 2011. Any combination of hierarchies can equally be used for backing-off.…”

Two-Stage Document Length Normalization for Information Retrieval

“…Interestingly, passage retrieval can also be viewed as two-stage normalization [Salton and Buckley 1991;Salton et al 1993;Callan 1994;Allan 1995;Mittendorf and Sch äuble 1994;Kaszkiel and Zobel 1997;Kaszkiel et al 1999;Liu and Croft 2002;Bendersky and Kurland 2008;Na et al 2008b;Lv and Zhai 2009b;Lv and Zhai 2010;Krikon et al 2010;Krikon and Kurland 2011]. Because scopes are more similar in passages themselves than in documents, using passages itself can be considered as a type of scope normalization.…”