Rios

Li, Youfu; Li, Mingda; Ding, Ling; Interlandi, Matteo

doi:10.1145/3267809.3267814

Cited by 12 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To enable CBO, either an associated Hive database has to be configured to store pre-computed table statistics or extra overhead is taken for statistics computation for each ad-hoc query. [66,67] design sophisticated cost models with available statistics for join order selection and cost estimation. [68] specifically optimizes the execution logic of key operators in Spark SQL including sort and aggregate.…”

Section: Query Optimization For Distributed Data Processing Systemsmentioning

confidence: 99%

“…Previous research has shown that directly applying models designed for RDBMSs does not yield satisfactory results for Spark SQL [70]. To optimize Spark SQL execution, early works [66,67] have attempted to create cost models using table and runtime statistics, which often fall short in accurately estimating the cost of complex queries. RAAL [70] was the first to design a deep learning (DL)-based cost estimation model for Spark SQL, aiming at providing better estimation accuracy for a given physical execution plan.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards advanced distributed data processing: framework, optimization, and application

Liu

View full text Add to dashboard Cite

The surge in available big data has drawn significant interest in distributed processing methods capable of handling the ever-expanding data volume and increasing computational complexities efficiently and at scale. While existing distributed data processing frameworks, such as Apache Spark, have proven effective in various applications, there is still considerable room for improvement and exploration in this field. This thesis focuses on three key aspects of advancing distributed data processing using Apache Spark. First, a novel framework is introduced to extend Spark's capabilities, enabling the efficient processing of large-scale spatio-temporal data to better serve machine-learning applications. This framework not only achieves high efficiency but also provides a user-friendly interface. Second, a deep-learning-based optimization approach tailored to enhance the efficiency of Spark SQL execution is proposed. The end-to-end system integration of this approach leads to practical performance gains. Last, a distributed solution for the computational-intensive large-scale microscopic crowd simulation is designed and implemented aiming to improve the scalability and efficiency of such applications. These three works collectively expand the application of distributed computing and enhance efficiency through the implementation of state-of-the-art techniques.

show abstract

Section: Query Optimization For Distributed Data Processing Systemsmentioning

confidence: 99%