Diversity in ranking via resistive graph centers

Abstract

Users can rarely reveal their information need in full detail to a search engine within 1-2 words, so search engines need to "hedge their bets" and present diverse results within the precious 10 response slots. Diversity in ranking is of much recent interest. Most existing solutions estimate the marginal utility of an item given a set of items already in the response, and then use variants of greedy set cover. Others design graphs with the items as nodes and choose diverse items based on visit rates (PageRank). Here we introduce a radically new and natural formulation of diversity as finding centers in resistive graphs. Unlike in PageRank, we do not specify the edge resistances (equivalently, conductances) and ask for node visit rates. Instead, we look for a sparse set of center nodes so that the effective conductance from the center to the rest of the graph has maximum entropy. We give a cogent semantic justification for turning PageRankthus on its head. In marked deviation from prior work, our edge resistances are learnt from training data. Inference and learning are NP-hard, but we give practical solutions. In extensive experiments with subtopic retrieval, social network search, and document summarization, our approach convincingly surpasses recently-published diversity algorithms like subtopic cover, max-marginal relevance (MMR), Grasshopper, DivRank, and SVMdiv.

Rel-Div: Generating Diversified Query Interpretations from Semantic Relations

“…In our experiments, we consider a couple of other approaches to diversification, which have been reported in literature, though used in other problem settings. These include variants of GCD [5] and Affinity Propagation [6,7]. M-Div : Uses page rank matrix M as in GCD instead of the C q matrix.…”

“…Most prior research has focused on generating diversified result [14,16,3,12,5,17,9,8,10,2,13,11]. Inspired by the work GCD [5] and MMR [4], we develop a new technique for diversity ranking of interpretations based on an interpretation graph. As part of this technique, we propose an algorithm to learn the node and edge weights of the interpretation graph iteratively in a biconvex optimization setting.…”

Affinity-Preserving Random Walk for Multi-Document Summarization

“…Also, the normalization in DivRank from affinity matrix to transition matrix is the same as PageRank. Another notable diversified graphbased ranking method GCD (Dubey et al, 2011) relies on large amounts of training data to learn edge conductances.…”

Latent dirichlet allocation based diversified retrieval for e-commerce search

“…Then the diverse items are selected based on different criterions, including the information richness [28], a reinforced random walk [15] and an absorbing random walk [30]. Instead of calculating the weight of the edge with some specific function, Dubey et al [8] learned the edge weights from data and then selected the center nodes that maximize the entropy of the conductance between the center nodes and the rest of the graph.…”