Early exit optimizations for additive machine learned ranking systems

Cambazoğlu, B. Barla; Zaragoza, Hugo; Chapelle, Olivier; Jiang, Chen; Liao, Ciya; Zheng, Zhaohui; Degenhardt, Jon

doi:10.1145/1718487.1718538

Cited by 102 publications

(105 citation statements)

References 26 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…We build ranking functions that satisfy each query's time requirement, whereas query evaluation strategies, such as early-termination techniques [7,2,28], simply improve the speed of existing ranking models, but do not provide an explicit mechanism for controlling the tradeoff between effectiveness and efficiency. The same critique applies to work on index pruning [8,23], where the goal is to improve query evaluation speed by reducing the size of the index.…”

Section: Related Workmentioning

confidence: 99%

Ranking under temporal constraints

Wang

Metzler

Lin

2010

Proceedings of the 19th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

This paper introduces the notion of temporally constrained ranked retrieval, which, given a query and a time constraint, produces the best possible ranked list within the specified time limit. Naturally, more time should translate into better results, but the ranking algorithm should always produce some results. This property is desirable from a number of perspectives: to cope with diverse users and information needs, as well as to better manage system load and variance in query execution times. We propose two temporally constrained ranking algorithms based on a class of probabilistic prediction models that can naturally incorporate efficiency constraints: one that makes independent feature selection decisions, and the other that makes joint feature selection decisions. Experiments on three different test collections show that both ranking algorithms are able to satisfy imposed time constraints, although the joint model outperforms the independent model in being able to deliver more effective results, especially under tight time constraints, due to its ability to capture feature dependencies.

show abstract

Section: Related Workmentioning

confidence: 99%

Ranking under temporal constraints

Wang

Metzler

Lin

2010

Proceedings of the 19th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…Examples of this search architecture abound in industry [7] and academia [8,6]. In fact, the TREC 2013 Microblog evaluation is exactly set up along these lines: participants do not have access to the raw collection-instead, they must complete the task via a search API that returns candidate results.…”

Section: Discussionmentioning

confidence: 99%

The Impact of Future Term Statistics in Real-Time Tweet Search

Wang

Lin

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In the real-time tweet search task operationalized in the TREC Microblog evaluations, a topic consists of a query Q and a time t, modeling the task where the user wishes to see the most recent but relevant tweets that address the information need. To simulate the real-time aspect of the task in an evaluation setting, many systems search over the entire collection and then discard results that occur after the query time. This approach, while computationally efficient, "cheats" in that it takes advantage of term statistics from documents not available at query time (i.e., future information). We show, however, that such results are nearly identical to a "gold standard" method that builds a separate index for each topic containing only those documents that occur before the query time. The implications of this finding on evaluation, system design, and user task models are discussed.

show abstract

“…Their applicability in practice is mainly limited by the fact that they have to evaluate many rankers at test time, and it is well known that the evaluation time is crucial in a real world learning-to-rank application. This motivates the development of a framework in which a controller can select the rankers to be evaluated based on the characteristics of individual queries (Cambazoglu et al 2010). Our future goal here is to model the problem as a Markov decision process, and solve it using standard reinforcement learning techniques (Dulac-Arnold et al 2011;Benbouzid et al 2011Benbouzid et al , 2012a.…”

Section: Discussionmentioning

confidence: 99%

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

et al. 2013

View full text Add to dashboard Cite

In subset ranking, the goal is to learn a ranking function that approximates a gold standard partial ordering of a set of objects (in our case, a set of documents retrieved for the same query). The partial ordering is given by relevance labels representing the relevance of documents with respect to the query on an absolute scale. Our approach consists of three simple steps. First, we train standard multi-class classifiers (AdaBoost.MH and multi-class SVM) to discriminate between the relevance labels. Second, the posteriors of multi-class classifiers are calibrated using probabilistic and regression losses in order to estimate the Bayes-scoring function which optimizes the Normalized Discounted Cumulative Gain (NDCG). In the third step, instead of selecting the best multi-class hyperparameters and the best calibration, we mix all the learned models in a simple ensemble scheme. Mach Learn (2013) 93:261-292 Our extensive experimental study is itself a substantial contribution. We compare most of the existing learning-to-rank techniques on all of the available large-scale benchmark data sets using a standardized implementation of the NDCG score. We show that our approach is competitive with conceptually more complex listwise and pairwise methods, and clearly outperforms them as the data size grows. As a technical contribution, we clarify some of the confusing results related to the ambiguities of the evaluation tools, and propose guidelines for future studies.

show abstract

Early exit optimizations for additive machine learned ranking systems

Cited by 102 publications

References 26 publications

Ranking under temporal constraints

Ranking under temporal constraints

The Impact of Future Term Statistics in Real-Time Tweet Search

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

Contact Info

Product

Resources

About