Delayed-Dynamic-Selective (DDS) Prediction for Reducing Extreme Tail Latency in Web Search

Abstract

A commercial web search engine shards its index among many servers, and therefore the response time of a search query is dominated by the slowest server that processes the query. Prior approaches target improving responsiveness by reducing the tail latency of an individual search server. They predict query execution time, and if a query is predicted to be long-running, it runs in parallel, otherwise it runs sequentially. These approaches are, however, not accurate enough for reducing a high tail latency when responses are aggregated from many servers because this requires each server to reduce a substantially higher tail latency (e.g., the 99.99th-percentile), which we call extreme tail latency.We propose a prediction framework to reduce the extreme tail latency of search servers. The framework has a unique set of characteristics to predict long-running queries with high recall and improved precision. Specifically, prediction is delayed by a short duration to allow many short-running queries to complete without parallelization, and to allow the predictor to collect a set of dynamic features using runtime information. These features estimate query execution time with high accuracy. We also use them to estimate the prediction errors to override an uncertain prediction by selectively accelerating the query for a higher recall.We evaluate the proposed prediction framework to improve search engine performance in two scenarios using a simulation study: (1) query parallelization on a multicore processor, and (2) query scheduling on a heterogeneous processor. The results show that, for both scenarios, the proposed framework is effective in reducing the extreme tail latency compared to a start-of-the-art predictor because of its higher recall, and it improves server throughput by more than 70% because of its improved precision.

Efficient & Effective Selective Query Rewriting with Efficiency Predictions

“…Dynamic pruning techniques such as WAND and BMW o er some relief as they o er the potential to safely skip the decompression of postings and the scoring of documents that cannot make the current top K. is makes the exact response time of a query di cult to predict, as not every posting in the postings lists will be decompressed and scored. Nevertheless recent work has considered making accurate predictions on the e ciency of a query, either in terms of absolute response time [29], or in terms of those queries with response times exceeding a threshold [19,21].…”

“…E ciency predictions facilitate a number of applications for ensuring e cient yet e ective retrieval -for instance, routing queries among busy replicated query shard servers [29]; selectively deploying multiple CPU cores for slow queries [19,21]; or adjusting the pruning aggressiveness or size of K for di erent queries [5,14,38]. Of these, the work of Tonello o et al [38] is among the most similar to ours, in that they vary the number of documents to be retrieved, K, as well as the pruning aggressiveness, before passing to a learning-to-rank re-ranking phase, based on the predicted execution time of the query.…”

“…As shown in [19,21,29,38], by using statistics derived from terms and queries it is possible to estimate the query processing time. However, since we must deal with ephemeral posting lists, most of the statistics used in prior research are not applicable.…”

“…Indeed, while recent work in query e ciency prediction has shown the possibility of estimating the execution time of a query prior to its processing [19,21,29,38], none of the existing work has considered the execution time of queries containing query operators that generate ephemeral posting lists, such as #syn, or #1. is makes it di cult to select among query rewriting strategies that use such operators, as their likely execution time is unknown. Hence, in this work, we study the cost of scoring ephemeral posting lists, and use these observations to de ne accurate query e ciency predictions for advanced query operators.…”

Few-to-Many

“…Moreover, reducing each server's tail latency is critical when a request spans several servers and responses are aggregated from these servers. In this case, the slower servers typically dominate the response time [22].…”