Search Result Diversification in Short Text Streams

Abstract

We consider the problem of search result diversification for streams of short texts. Diversifying search results in short text streams is more challenging than in the case of long documents, as it is difficult to capture the latent topics of short documents. To capture the changes of topics and the probabilities of documents for a given query at a specific time in a short text stream, we propose a dynamic Dirichlet multinomial mixture topic model, called D2M3, as well as a Gibbs sampling algorithm for the inference. We also propose a streaming diversification algorithm, SDA, that integrates the information captured by D2M3 with our proposed modified version of the PM-2 (Proportionality-based diversification Methodsecond version) diversification algorithm. We conduct experiments on a Twitter dataset and find that SDA statistically significantly outperforms state-of-the-art non-streaming retrieval methods, plain streaming retrieval methods, as well as streaming diversification methods that use other dynamic topic models.

Revisiting the cluster-based paradigm for implicit search result diversification

“…The work by Liang et al [26] showed that fusing results of different rankers does aid diversification. Moreover, Liang et al [28] also explored how to perform search result diversification for streams of short texts (e.g., Twitter messages). The experimental results show that diversification for streams of short texts is quite different from diversification for long documents, and specific models have to be carefully designed.…”

Collaboratively Tracking Interests for User Clustering in Streams of Short Texts

“…Since the well-known topic models, probabilistic latent semantic indexing [12] and LDA (Latent Dirichlet Allocation) [7], were proposed, topic models with dynamics have been widely studied. These include the Dynamic Topic Model (DTM) [13], Dynamic Mixture Model (DMM) [14], Topic over Time (ToT) [15], Topic Tracking Model (TTM) [16], infinite topic-cluster model [17], and more recently, generalized dynamic topic model [18], dynamic User Clustering Topic model (UCT) [6,10], dynamic topic model for search diversification [19], Dynamic Clustering Topic model (DCT) [20] and scaling-up dynamic model [21]. All of these models except DCT aim at inferring documents' dynamic topic distributions rather than user clustering.…”

Collaborative, Dynamic and Diversified User Profiling

“…Since the well-known topic models, PLSI (Probabilistic Latent Semantic Indexing) (Hofmann 1999) and LDA (Latent Dirichlet Allocation) (Blei, Ng, and Jordan 2003), were proposed, topic models with dynamics have been widely studied. These include Dynamic Topic Model (Blei and Lafferty 2006), Dynamic Mixture Model (Wei, Sun, and Wang 2007), Topic over Time (Wang and McCallum 2006), Topic Tracking Model (Iwata et al 2009), and more recently, dynamic Dirichlet multinomial mixture topic model (Liang et al 2017c), user expertise tracking topic model (Liang 2018) and user collaborative interest tracking topic model (Liang, Yilmaz, and Kanoulas 2018). To our knowledge, none of existing dynamic topic models has considered the problem of user profiling for short texts that utilizes collaborative information to infer topic distributions.…”

“…x is a Digamma function. Our derivations of the update rules for α t , β t and γ t in (6) are analogous to those in (Liang, Yilmaz, and Kanoulas 2018;Liang et al 2017c;2017b).…”

“…After we obtain θ t,u , ψ t,u and φ t,z , inspired by PM-2 diversification method (Dang and Croft 2012), we closely follow the work in (Liang et al 2017c;2018) and propose a streaming keyword diversification model (i.e., Algorithm 2), SKDM. To generate top-k diversified keywords for each user u at t, SKDM starts with an empty keyword set w t,u with k empty seats (step 2 of Algorithm 2), and a set of candidate keywords (step 3), v, which is the whole words v in the vocabulary, i.e., initially let v = v. For each of the seats, it computes the quotient qt[z|t, u] for each topic z given a user u at t by the Sainte-Laguë formula (step 9): qt[z|t, u] = δt,u,z 2s z|t,u +1 , where δ t,u,z is the final probability of the user u has interest on topic z at t and is set to be δ t,u,z = (1 − λ)P (z|t, u) + λP (z|t, f t,u ) (step 5), and s z|t,u is the "number" of seats occupied by topic z (in initialization, let s z|t,u = 0 for all topics (step 6)).…”