Entropy-based histograms for selectivity estimation

To, Hien; Chiang, Kuorong; Shahabi, Cyrus

doi:10.1145/2505515.2505756

Cited by 24 publications

(15 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other histogram-based cardinality prediction methods utilize wavelets [31] or entropy-based [28]; the list is not exhausted. Briefly, the idea is to apply wavelet decomposition to the dataset to obtain a compact data synopsis based on the wavelet coefficients.…”

Section: Related Workmentioning

confidence: 99%

Scalable aggregation predictive analytics

2017

View full text Add to dashboard Cite

We introduce a predictive modeling solution that provides high quality predictive analytics over aggregation queries in Big Data environments. Our predictive methodology is generally applicable in environments in which large-scale data owners may or may not restrict access to their data and allow only aggregation operators like COUNT to be executed over their data. In this context, our methodology is based on historical queries and their answers to accurately predict ad-hoc queries' answers. We focus on the widely used set-cardinality, i.e., COUNT, aggregation query, as COUNT is a fundamental operator for both internal data system optimizations and for aggregation-oriented data exploration and predictive analytics. We contribute a novel, query-driven Machine Learning (ML) model whose goals are to: (i) learn the query-answer space from past issued queries, (ii) associate the query space with local linear regression & associative function estimators, (iii) define query similarity, and (iv) predict the cardinality of the answer set of unseen incoming queries, referred to the Set Cardinality Prediction (SCP) problem. Our ML model incorporates incremental ML algorithms for ensuring high quality prediction results. The significance of contribution lies in that it (i) is the only query-driven solution applicable over general Big Data environments, which include restrictedaccess data, (ii) offers incremental learning adjusted for Christos Anagnostopoulos christos.anagnostopoulos@glasgow.ac.uk Peter Triantafillou peter.triantafillou@glasgow.ac.uk 1 School of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK arriving ad-hoc queries, which is well suited for querydriven data exploration, and (iii) offers a performance (in terms of scalability, SCP accuracy, processing time, and memory requirements) that is superior to data-centric approaches. We provide a comprehensive performance evaluation of our model evaluating its sensitivity, scalability and efficiency for quality predictive analytics. In addition, we report on the development and incorporation of our ML model in Spark showing its superior performance compared to the Spark's COUNT method.

show abstract

Section: Related Workmentioning

confidence: 99%

Scalable aggregation predictive analytics

2017

View full text Add to dashboard Cite

show abstract

“…Fundamentally, the limitations in STHs in our problem stem from the fact that they estimate p(x|y, q), thus, having to data access (in m-d STHs, at least one scan of the set B is required), deal with the underlying data distribution and make certain assumptions of the statistical dependencies of data. Other histogram-based SCP methods utilize wavelets [11], singular value decomposition [12], value transformations [13], and entropy-based [16]; the list is not exhausted. Briefly, the idea is to apply wavelet decomposition to the dataset to obtain a compact data synopsis based on the wavelet coefficients.…”

Section: Introductionmentioning

confidence: 99%

Learning to accurately COUNT with query-driven predictive analytics

Anagnostopoulos

Triantafillou

2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Abstract-We study a novel solution to executing aggregation (and specifically COUNT) queries over large-scale data. The proposed solution is generally applicable, in the sense that it can be deployed in environments in which data owners may or may not restrict access to their data and allow only 'aggregation operators' to be executed over their data. For this, it is based on predictive analytics, driven by queries and their results. We propose a machine learning (ML) framework for the task (which can be adapted for different aggregates as well). We focus on the widely used set-cardinality (i.e., COUNT) aggregation operator, as it is a fundamental operator for both internal data system optimisations and for aggregation-query analytics. We contribute a novel, query-driven ML model whose goals are to: (i) learn the query space (access patterns), (ii) associate (complex) aggregation queries with the cardinality of their results, (iii) define query similarity and use it to predict the cardinality of the answer set of an ad-hoc incoming query. Our ML model incorporates incremental learning algorithms for ensuring high prediction accuracy even when both the querying patterns and the underlying data change. The significance of contribution lies in that it (i) is the only query-driven solution applicable over general environments which include restrictedaccess data, (ii) offers incremental learning adjusted for arriving ad-hoc queries, which is well suited for big data analytics, and (iii) offers a performance (in terms of prediction accuracy and time, and memory requirements) that is superior to datacentric approaches. We provide a comprehensive performance evaluation of our model, evaluating its sensitivity and comparative advantages versus acclaimed data-centric methods (self-tuning histograms, sampling, and multidimensional histograms).

show abstract

“…They summarize the multi-dimensional histograms and kernel density estimators. After 2003, the work in histograms can be divided into three categories: (1) fast algorithm for histogram construction [1,28,32,33,42]; (2) new partition methods to divide the data into different buckets to achieve better accuracy [14,58,88];…”

Section: Histogrammentioning

confidence: 99%

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

2021

View full text Add to dashboard Cite

Query optimizer is at the heart of the database systems. Cost-based optimizer studied in this paper is adopted in almost all current database systems. A cost-based optimizer introduces a plan enumeration algorithm to find a (sub)plan, and then uses a cost model to obtain the cost of that plan, and selects the plan with the lowest cost. In the cost model, cardinality, the number of tuples through an operator, plays a crucial role. Due to the inaccuracy in cardinality estimation, errors in cost model, and the huge plan space, the optimizer cannot find the optimal execution plan for a complex query in a reasonable time. In this paper, we first deeply study the causes behind the limitations above. Next, we review the techniques used to improve the quality of the three key components in the cost-based optimizer, cardinality estimation, cost model, and plan enumeration. We also provide our insights on the future directions for each of the above aspects.

show abstract

Entropy-based histograms for selectivity estimation

Cited by 24 publications

References 21 publications

Scalable aggregation predictive analytics

Scalable aggregation predictive analytics

Learning to accurately COUNT with query-driven predictive analytics

A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration

Contact Info

Product

Resources

About