A Resource-Aware Deep Cost Model for Big Data Query Processing

Li, Yan; Wang, Liwei; Wang, Sheng; Sun, Yuan; Peng, Zhiyong

doi:10.1109/icde53745.2022.00071

Cited by 4 publications

(7 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In distributed query processing systems, the execution plans are more complicated and dynamic, which is harder to model compared with conventional relational databases, leading to fewer existing studies in this area. RAAL [70] claims that the DL-based cost estimation models for RDBMSs cannot directly apply to distributed data systems and designs a resourceaware LSTM-based model to predict the cost of Spark physical plans. It assumes that the available resources change over time and adds a resource-attention layer to capture the change.…”

Section: Query Optimization For Distributed Data Processing Systemsmentioning

confidence: 99%

“…, v m ] ∈ R m×(m+n) as the complete embedding of a plan, where v i = (v s i , v o i ). Compared to RAAL [70] which applies the word2vec technique to all information in an operation statement, our categorical embedding method digs contextual information from the plan and mixes the utilization of simple one-hot and learned uSIF methods to reduce the model complexity and embedding space.…”

Section: Structure Embeddingmentioning

confidence: 99%

“…Base model. As a widely-used variant of Recurrent Neural Networks (RNNs), LSTM networks [165] have been proven effective in modeling query plans due to their advantages in learning long-term dependency over sequences with variable lengths [56,70]. Although a plan is executed top-down, the subsequent operations may influence the physical operator selection of the previous operations (e.g., indexing is more favored when being repeatedly used in subsequent operations).…”

Section: Deep Regression Modelmentioning

confidence: 99%

“…Modeling query performance in distributed environments presents a greater challenge than in RDBMSs as the physical operators are more complex to model, yet have a substantial influence on the execution efficiency. Previous research has shown that directly applying models designed for RDBMSs does not yield satisfactory results for Spark SQL [70]. To optimize Spark SQL execution, early works [66,67] have attempted to create cost models using table and runtime statistics, which often fall short in accurately estimating the cost of complex queries.…”

Section: Introductionmentioning

confidence: 99%

“…To optimize Spark SQL execution, early works [66,67] have attempted to create cost models using table and runtime statistics, which often fall short in accurately estimating the cost of complex queries. RAAL [70] was the first to design a deep learning (DL)-based cost estimation model for Spark SQL, aiming at providing better estimation accuracy for a given physical execution plan. However, to the best of our knowledge, all existing works focus merely on cost estimation and do not address the integration of the DL model into the Spark runtime, which is a non-trivial effort.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Towards advanced distributed data processing: framework, optimization, and application

Liu

View full text Add to dashboard Cite

The surge in available big data has drawn significant interest in distributed processing methods capable of handling the ever-expanding data volume and increasing computational complexities efficiently and at scale. While existing distributed data processing frameworks, such as Apache Spark, have proven effective in various applications, there is still considerable room for improvement and exploration in this field. This thesis focuses on three key aspects of advancing distributed data processing using Apache Spark. First, a novel framework is introduced to extend Spark's capabilities, enabling the efficient processing of large-scale spatio-temporal data to better serve machine-learning applications. This framework not only achieves high efficiency but also provides a user-friendly interface. Second, a deep-learning-based optimization approach tailored to enhance the efficiency of Spark SQL execution is proposed. The end-to-end system integration of this approach leads to practical performance gains. Last, a distributed solution for the computational-intensive large-scale microscopic crowd simulation is designed and implemented aiming to improve the scalability and efficiency of such applications. These three works collectively expand the application of distributed computing and enhance efficiency through the implementation of state-of-the-art techniques.

show abstract

Section: Query Optimization For Distributed Data Processing Systemsmentioning

confidence: 99%

Section: Structure Embeddingmentioning

confidence: 99%

Section: Deep Regression Modelmentioning

confidence: 99%