Fusion helps diversification

Abstract

A popular strategy for search result diversification is to first retrieve a set of documents utilizing a standard retrieval method and then rerank the results. We adopt a different perspective on the problem, based on data fusion. Starting from the hypothesis that data fusion can improve performance in terms of diversity metrics, we examine the impact of standard data fusion methods on result diversification. We take the output of a set of rankers, optimized for diversity or not, and find that data fusion can significantly improve state-of-the art diversification methods. We also introduce a new data fusion method, called diversified data fusion, which infers latent topics of a query using topic modeling, without leveraging outside information. Our experiments show that data fusion methods can enhance the performance of diversification and DDF significantly outperforms existing data fusion methods in terms of diversity metrics.

Search Result Diversification in Short Text Streams

“…He et al [18] propose a result diversification framework based on query-specific clustering and cluster ranking, in which diversification is restricted to documents belonging to clusters that potentially contain a high percentage of relevant documents. More recent implicit work includes set-based recommendation of diverse articles [1], term-level diversification [14], diversified data fusion [26], and neural-network-based diversification model [46]. Abbar et al [1] address the problem of providing diverse news recommendations related to an input article by leveraging user-generated data to refine lists of related articles.…”

“…Instead of trying to recover the topics for an ambiguous query, Dang and Croft [14] propose to use a simple greedy multi-document summarization algorithm for identifying topic terms for search result diversification from the initial ranking of documents. Liang et al [26] start from the hypothesis that data fusion can improve performance in terms of diversity metrics, examine the impact of standard data fusion methods on search result diversification, and propose a diversified data fusion algorithm to infer latent topics of a query using topic modeling model for diversification. Xia et al [46] propose to model the novelty of a document with a neural tensor network and learn a nonlinear novelty function based on the preliminary representation of the candidate document and other documents for diversification.…”

“…From the first work on topic models, the Probabilistic LSI model [19], they have received significant attention [4,17] and have been used in many retrieval tasks [26,43].…”

“…LDA represents each document as a finite mixture over "latent" topics where each topic is represented as a finite mixture over words in that document. Based on LDA, many extensions have been proposed, for example, to handle users' connections with particular documents and topics [36], to learn relations among different topics [23,24], for topics over time [42], for ad hoc retrieval [43], or for rank aggregation [26]. LDA has also been extended to clustering [48] and tweet summarization [35].…”

Diversity, Intent, and Aggregated Search

“…In the talk I will highlight recent developments and point out directions for future work. In particular, concerning search result diversification I will run through a new perspective on the problem by casting it as a data fusion problem, following [5], and inferring latent topics of the query for which the result set is being diversified. A second important algorithmic development concerns recent work on personalized search result diversification, based on structured learning [6].…”