Towards predicting query execution time for concurrent and dynamic database workloads

Abstract

Predicting query execution time is crucial for many database management tasks including admission control, query scheduling, and progress monitoring. While a number of recent papers have explored this problem, the bulk of the existing work either considers prediction for a single query, or prediction for a static workload of concurrent queries, where by "static" we mean that the queries to be run are fixed and known. In this paper, we consider the more general problem of dynamic concurrent workloads. Unlike most previous work on query execution time prediction, our proposed framework is based on analytic modeling rather than machine learning. We first use the optimizer's cost model to estimate the I/O and CPU requirements for each pipeline of each query in isolation, and then use a combination queueing model and buffer pool model that merges the I/O and CPU requests from concurrent queries to predict running times. We compare the proposed approach with a machine-learning based approach that is a variant of previous work. Our experiments show that our analytic-model based approach can lead to competitive and often better prediction accuracy than its machine-learning based counterpart.

Resource bricolage for parallel database systems

“…However, it cannot provide accurate estimates for queries running on database systems [17]. On the other hand, the second category of work focuses on query running time prediction before a query starts [4,11,13,31,32]. In [32] the authors proposed a technique to calibrate the cost units in the optimizer cost model to match the true performance of the hardware and software on which the query will be run, in order to estimate query execution time.…”

A proposed method for handling queries of mixed workloads in databases

“…These progress indicators [4] help in identifying long running queries at an early stage so as to prevent our workload from negative impact. Luo et al [8] modified the existing indicators to find the remaining execution time for a query running concurrently with other queries.. Wu et al [5] also contributed towards predicting execution time of dynamic and concurrent workload.…”

Distribution-based query scheduling

“…Solutions proposed so far cover both OLTP [12] and OLAP [22,36,35] queries, as well as both stand-alone [12,36] and concurrent [10,35] queries. Because most of these solutions are based on machine-learning approaches, and because real data contain noise and inherent uncertainties, it is reasonable to assume there exists unavoidable imprecision in the predicted query execution time.…”

“…Despite a great deal of decade-old and recent effort [12,10,22,36,35], accurate prediction of query running times for dynamic, concurrent workloads remains largely elusive.…”