Evaluation strategies for top-k queries over memory-resident inverted indexes

Abstract

Top-k retrieval over main-memory inverted indexes is at the core of many modern applications: from large scale web search and advertising platforms, to text extenders and content management systems. In these systems, queries are evaluated using two major families of algorithms: document-at-a-time (DAAT) and term-at-a-time (TAAT). DAAT and TAAT algorithms have been studied extensively in the research literature, but mostly in disk-based settings. In this paper, we present an analysis and comparison of several DAAT and TAAT algorithms used in Yahoo!'s production platform for online advertising. The low-latency requirements of online advertising systems mandate memory-resident indexes. We compare the performance of several query evaluation algorithms using two real-world ad selection datasets and query workloads. We show how some adaptations of the original algorithms for main memory setting have yielded significant performance improvement, reducing running time and cost of serving by 60% in some cases. In these results both the original and the adapted algorithms have been evaluated over memory-resident indexes, so the improvements are algorithmic and not due to the fact that the experiments used main memory indexes.

Boosting Search Performance Using Query Variations

“…These approaches tend to be more efficient than are DaaT MaxScore and term-at-a-time (TaaT) approaches [67,68], particularly for short queries, the most common scenario in web search. However, for long queries or large candidate sets, the case is less clear-cut [23,33,54]; moreover, fusion over query variations often leads to very long queries.…”

Speeding up Extreme Multi-Label Classifier by Approximate Nearest Neighbor Search

“…However, if a few n j corresponding to "head" labels are near N, which means almost all data points have the same label, the above cost reaches O(N 2 ). Efficient algorithms for top-k retrieval on an inverted index [11] or finding approximate k-nearest neighbors [12] can be applied to this situation. However, we focus on tail labels and ignore head labels in some cases.…”

WHInter: A Working set algorithm for High-dimensional sparse second order Interaction models

“…The interest of casting these updates as variants of MIPS problems is to exploit the ideas developed in the literature for solving these problems efficiently. Teflioudi and Gemulla (2016) and Fontoura et al (2011) give good overviews of MIPS solvers developed for recommender systems and information retrieval applications respectively. In both cases, the proposed methods rely on two main ideas: (i) adequate indexing techniques or data structures and (ii) pruning criteria which allow to not compute all inner products entirely.…”