NeuroCard

Abstract

Query optimizers rely on accurate cardinality estimates to produce good execution plans. Despite decades of research, existing cardinality estimators are inaccurate for complex queries, due to making lossy modeling assumptions and not capturing inter-table correlations. In this work, we show that it is possible to learn the correlations across all tables in a database without any independence assumptions. We present NeuroCard, a join cardinality estimator that builds a single neural density estimator over an entire database. Leveraging join sampling and modern deep autoregressive models, NeuroCard makes no inter-table or inter-column independence assumptions in its probabilistic modeling. NeuroCard achieves orders of magnitude higher accuracy than the best prior methods (a new state-of-the-art result of 8.5x maximum error on JOB-light), scales to dozens of tables, while being compact in space (several MBs) and efficient to construct or update (seconds to minutes).

Representative Routes Discovery from Massive Trajectories

“…In the rest of this paper, we use distinctiveness estimation and MCE interchangeably. Notably, when has only one query (| | = 1) and contains only one dataset (| | = 1), the distinctiveness estimation is equivalent to the well-known SCE problem [39,56,60].…”

“…6.1, two implementations were considered -our distinctiveness estimation method DE, and the state-of-the-art cardinality estimation method IRIS. We compare the effectiveness using q-error, which is widely used in other cardinality estimation studies [30,39,56,57]. Using the default setting, we estimate the distinctiveness for each dataset in , and then compute the q-error for all of the datasets.…”

“…Query-driven methods [17,22,31,41,46,47,49,52,53] deploy discriminative models trained on a set of historic queries to predict the cardinality for a single query. Data-driven methods [24,37,39,51,54,56,57,60] deploy generative models trained on data without using query workloads. In general, query-driven methods are inflexible, especially when representative queries are unavailable, and data-driven methods can achieve better performance than query-driven methods [21].…”

“…a query set. Specifically, it is difficult to identify the overlap between the results for a query set on multiple datasets -a problem that has never been addressed in the existing literature [22,24,39,56,60].…”

“…We call this CE problem Multi-Query-Dataset Cardinality Estimation (MCE). The MCE problem differs from the existing CE problem [21,24,30,39,56,60], which estimates the cardinality using a single query and a single dataset. To better distinguish these two problems, we call the latter Single-Query-Dataset Cardinality Estimation (SCE).…”