Term level search result diversification

Abstract

Current approaches for search result diversification have been categorized as either implicit or explicit. The implicit approach assumes each document represents its own topic, and promotes diversity by selecting documents for different topics based on the difference of their vocabulary. On the other hand, the explicit approach models the set of query topics, or aspects. While the former approach is generally less effective, the latter usually depends on a manually created description of the query aspects, the automatic construction of which has proven difficult. This paper introduces a new approach: term-level diversification. Instead of modeling the set of query aspects, which are typically represented as coherent groups of terms, our approach uses terms without the grouping. Our results on the ClueWeb collection show that the grouping of topic terms provides very little benefit to diversification compared to simply using the terms themselves. Consequently, we demonstrate that term-level diversification, with topic terms identified automatically from the search results using a simple greedy algorithm, significantly outperforms methods that attempt to create a full topic structure for diversification.

Search Result Diversification in Short Text Streams

“…The goal is to produce a more-diverse ranked list with respect to a set of aspects associated with the query [2,13,14]. Search result diversification is similar to ad hoc search but differs in its judgment criteria and evaluation measures [38].…”

“…He et al [18] propose a result diversification framework based on query-specific clustering and cluster ranking, in which diversification is restricted to documents belonging to clusters that potentially contain a high percentage of relevant documents. More recent implicit work includes set-based recommendation of diverse articles [1], term-level diversification [14], diversified data fusion [26], and neural-network-based diversification model [46]. Abbar et al [1] address the problem of providing diverse news recommendations related to an input article by leveraging user-generated data to refine lists of related articles.…”

“…They explore different diversity distances that rely on the content of user comments on articles such as sentiments and entities. Instead of trying to recover the topics for an ambiguous query, Dang and Croft [14] propose to use a simple greedy multi-document summarization algorithm for identifying topic terms for search result diversification from the initial ranking of documents. Liang et al [26] start from the hypothesis that data fusion can improve performance in terms of diversity metrics, examine the impact of standard data fusion methods on search result diversification, and propose a diversified data fusion algorithm to infer latent topics of a query using topic modeling model for diversification.…”

“…Explicitly computing the probabilities of aspects of a query can improve diversification performance [2,14,37]. Following Ref.…”

Extending Faceted Search to the Open-Domain Web

“…search. Search result diversification has been studied as a method of tackling ambiguous or multi-faceted queries while a ranked list of documents remains the primary output feature of web search engine today (Agrawal et al, 2009;Clarke et al, 2008;Santos et al, 2010;Dang and Croft, 2013). The purpose is to diversify the ranked list to account for different search intents or query subtopics.…”