Utilizing passage-based language models for ad hoc document retrieval

Abstract

To cope with the fact that, in the ad hoc retrieval setting, documents relevant to a query could contain very few (short) parts (passages) with query-related information, researchers proposed passage-based document ranking approaches. We show that several of these retrieval methods can be understood, and new ones can be derived, using the same probabilistic model. We use language-model estimates to instantiate specific retrieval algorithms, and in doing so present a novel passage language model that integrates information from the containing document to an extent controlled by the estimated document homogeneity. Several document-homogeneity measures that we present yield passage language models that are more effective than the standard passage model for basic document retrieval and for constructing and utilizing passage-based relevance models; these relevance models also outperform a document-based relevance model. Finally, we demonstrate the merits in using the document-homogeneity measures for integrating document-query and passage-query similarity information for document retrieval.Keywords Ad hoc document retrieval Á Passage-based language models Á Document homogeneity Á Relevance models Á Passage-based relevance models Portions of this work appeared in Bendersky and Kurland (2008b) and Bendersky (2007).

A passage-based approach to learning to rank documents

“…The line of work most related to ours is on passage-based document retrieval [3,4,7,13,21,24,25,28,30,32,41,44,[53][54][55]. As already noted, the most commonly used passage-based document retrieval methods are ranking a document by the maximum querysimilarity of its passages [4,7,24,25,32,44,55] and by interpolating this similarity with the document-query similarity [4,7,44,55]. We show that our best-performing methods substantially outperform a highly effective method that integrates document-query and passage-query similarities [4].…”

“…As a result, there has been a large body of work on passage-based document retrieval: utilizing information induced from document passages to rank the documents; e.g., [4,7,25,32,55]. The most commonly used passage-based document retrieval methods rank a document by the highest query similarity exhibited by any of its passages [4,7,25,32,55] and by integrating this similarity with the document-query similarity [4,7,55].…”

Enriching document representation via translation for improved monolingual information retrieval

“…Expansion terms are selected from hand-crafted thesauri such as WordNet [10], co-occurrence based similarity thesauri [15], highly-ranked retrieved documents (i.e., pseudorelevance feedback) [23,43], highly-ranked retrieved passages [2,26], or external collections such as the Web or Wikipedia [9,42]. Document expansion has a similar motivation as query expansion, but expansion is applied to documents and not to the query [24,21].…”

“…To deal with these problems, many studies in IR have investigated word sense disambiguation (WSD) on queries and documents [20,40,34,35,13,29,36,16], or have performed query expansion [15,23,43,9,10,42,2,26] or document expansion [3,24,21] by appending semantically related words into the original query or document. Some of these approaches (e.g., pseudo-relevance feedback) have shown marked improvements in retrieval performance.…”

Utilizing local evidence for blog feed search

“…One of the effective approaches is passage retrieval, in which the relevance score of a document is boosted by an additional score estimated using passage-level evidence. Passage retrieval has turned out to significantly improve the baseline using only traditional document-level evidence (Callan 1994;Kaszkiel and Zobel 1997;Kaszkiel and Zobel 2001;Salton et al 1993;Na et al 2008b;Bendersky and Kurland 2010).…”