FSPN: A New Class of Probabilistic Graphical Model

Wu, Ziniu; Zhu, Rong; Pfadler, Andreas; Han, Yuxing; Li, Jiangneng; Qian, Zhengping; Zeng, Kai; Zhou, Jingren

doi:10.48550/arxiv.2011.09020

Cited by 4 publications

(4 citation statements)

References 20 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In traditional methods, histogram and multiple sampling-based CardEst methods could be used to count distinct number of values. For ML-based methods, we find that the SPN model [24] and FSPN model [38] could also support distinct count with small adaptions. We elaborate the details as follows.…”

Section: Glue For Distinct Countmentioning

confidence: 98%

Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size

Zhu¹,

Zeng²,

Pfadler³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Cardinality estimation (CardEst), a central component of the query optimizer, plays a significant role in generating high-quality query plans in DBMS. The CardEst problem has been extensively studied in the last several decades, using both traditional and ML-enhanced methods. Whereas, the hardest problem in CardEst, i.e., how to estimate the join query size on multiple tables, has not been extensively solved. Current methods either reply on independence assumptions or apply techniques with heavy burden, whose performance is still far from satisfactory. Even worse, existing CardEst methods are often designed to optimize one goal, i.e., inference speed or estimation accuracy, which can not adapt to different occasions.In this paper, we propose a very general framework, called G , to tackle with these challenges. Its key idea is to elegantly decouple the correlations across different tables and losslessly merge single table CardEst results to estimate the join query size. G supports obtaining the single table-wise CardEst results using any existing CardEst method and can process any complex join schema. Therefore, it easily adapts to different scenarios having different performance requirements, i.e., OLTP with fast estimation time or OLAP with high estimation accuracy. Meanwhile, we show that G can be seamlessly integrated into the plan search process and is able to support counting distinct number of values. All these properties exhibit the potential advances of deploying G in real-world DBMS.

show abstract

Section: Glue For Distinct Countmentioning

confidence: 98%

Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size

Zhu¹,

Zeng²,

Pfadler³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…13) FLAT [69], based on factorize-split-sum-product networks (FSPN) [63], improves over SPN by adaptively decomposing ( ) according to the attribute dependence level. It adds the factorize node to split as ( ) • ( | ) where and are highly and weakly correlated attributes in .…”

Section: Ml-basedmentioning

confidence: 99%

Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation

Han,

Wu,

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Cardinality estimation (CardEst) plays a significant role in generating high-quality query plans for a query optimizer in DBMS. In the last decade, an increasing number of advanced CardEst methods (especially ML-based) have been proposed with outstanding estimation accuracy and inference latency. However, there exists no study that systematically evaluates the quality of these methods and answer the fundamental problem: to what extent can these methods improve the performance of query optimizer in real-world settings, which is the ultimate goal of a CardEst method.In this paper, we comprehensively and systematically compare the effectiveness of CardEst methods in a real DBMS. We establish a new benchmark for CardEst, which contains a new complex real-world dataset STATS and a diverse query workload STATS-CEB. We integrate multiple most representative CardEst methods into an open-source database system PostgreSQL, and comprehensively evaluate their true effectiveness in improving query plan quality, and other important aspects affecting their applicability, ranging from inference latency, model size, and training time, to update efficiency and accuracy. We obtain a number of key findings for the CardEst methods, under different data and query settings. Furthermore, we find that the widely used estimation accuracy metric (Q-Error) cannot distinguish the importance of different sub-plan queries during query optimization and thus cannot truly reflect the query plan quality generated by CardEst methods. Therefore, we propose a new metric P-Error to evaluate the performance of CardEst methods, which overcomes the limitation of Q-Error and is able to reflect the overall end-to-end performance of CardEst methods. We have made all of the benchmark data and evaluation code publicly available at h ps://github.com/Nathaniel-Han/End-to-End-CardEst-Benchmark.

show abstract

“…First, the encoder for single table is trained in a self-supervised fashion, which can be costly and can hardly benefit from meta-learning. Thus, an efficient unsupervised approach (such as DAR [8,26], SPN [30], FSPN [40], BN [9]) can be used to alleviate the training cost. Second, the optimal join order for a query with a large number of tables is very expensive to obtain, limiting the MTF-QO's ability to extrapolate to very complex queries.…”

Section: Research Opportunitiesmentioning

confidence: 99%

A Unified Transferable Model for ML-Enhanced DBMS

Wu¹,

Yan²,

Yu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, the database management system (DBMS) community has witnessed the power of machine learning (ML) solutions for DBMS tasks. Despite their promising performance, these existing solutions can hardly be considered satisfactory. First, these MLbased methods in DBMS are not effective enough because they are optimized on each specific task, and cannot explore or understand the intrinsic connections between tasks. Second, the training process has serious limitations that hinder their practicality, because they need to retrain the entire model from scratch for a new DB. Moreover, for each retraining, they require an excessive amount of training data, which is very expensive to acquire and unavailable for a new DB. We propose to explore the transferabilities of the ML methods both across tasks and across DBs to tackle these fundamental drawbacks.In this paper, we propose a unified model MTMLF that uses a multi-task training procedure to capture the transferable knowledge across tasks and a pre-train fine-tune procedure to distill the transferable meta knowledge across DBs. We believe this paradigm is more suitable for cloud DB service, and has the potential to revolutionize the way how ML is used in DBMS. Furthermore, to demonstrate the predicting power and viability of MTMLF, we provide a concrete and very promising case study on query optimization tasks. Last but not least, we discuss several concrete research opportunities along this line of work.

show abstract

FSPN: A New Class of Probabilistic Graphical Model

Cited by 4 publications

References 20 publications

Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size

Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size

Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation

A Unified Transferable Model for ML-Enhanced DBMS

Contact Info

Product

Resources

About