Performance Assurance Model for Applications on SPARK Platform

Singhal, Rekha; Singh, Praveen Kumar

doi:10.1007/978-3-319-72401-0_10

Cited by 21 publications

(18 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Spark job execution consists of multiple parallel or sequential stages and each stage is composed of multiple tasks. The performance of each task is influenced by several factors at each task phase [46,103].…”

Section: Batch Processing Systemsmentioning

confidence: 99%

“…However, system performance is directly linked to a vast array of configuration parameters, which control various aspects of system execution, ranging from low-level memory settings and thread counts to higher-level decisions such as 43:2 H. Herodotou et al resource management and load balancing [88]. Improper settings of configuration parameters are shown to have detrimental effects on the overall system performance and stability [39,61,103].…”

Section: Introductionmentioning

confidence: 99%

“…(1) Large and complex parameter space: Hadoop, Spark, and Storm have over 200 configurable parameters each [22,70,103]. To make matters worse, some parameters might affect the performance of different jobs in different ways, while certain groups of parameters may have dependent effects (i.e., a good setting for one parameter may depend on the setting of a different parameter) [61,66].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotou

Chen

2020

ACM Comput. Surv.

View full text Add to dashboard Cite

Big data processing systems (e.g., Hadoop, Spark, Storm) contain a vast number of configuration parameters controlling parallelism, I/O behavior, memory settings, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators grapple with understanding and tuning them to achieve good performance. We investigate existing approaches on parameter tuning for both batch and stream data processing systems and classify them into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We summarize the pros and cons of each approach and raise some open research problems for automatic parameter tuning.

show abstract

Section: Batch Processing Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotou

Chen

2020

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…For establishing the task execution cost model in Spark, we improve the method proposed by Singhal and Singh [15] and add the cost generated by sorting operation. When calculating the Stage cost, reading input data, merging and sorting intermediate data, and writing output data are considered, that is…”

Section: Cost Modelmentioning

confidence: 99%

Query Execution Optimization in Spark SQL

Zhao

Zhai

et al. 2020

Scientific Programming

View full text Add to dashboard Cite

Spark SQL is a big data processing tool for structured data query and analysis. However, due to the execution of Spark SQL, there are multiple times to write intermediate data to the disk, which reduces the execution efficiency of Spark SQL. Targeting on the existing issues, we design and implement an intermediate data cache layer between the underlying file system and the upper Spark core to reduce the cost of random disk I/O. By using the query pre-analysis module, we can dynamically adjust the capacity of cache layer for different queries. And the allocation module can allocate proper memory for each node in cluster. According to the sharing of the intermediate data in the Spark SQL workflow, this paper proposes a cost-based correlation merging algorithm, which can effectively reduce the cost of reading and writing redundant data. This paper develops the SSO (Spark SQL Optimizer) module and integrates it into the original Spark system to achieve the above functions. This paper compares the query performance with the existing Spark SQL by experiment data generated by TPC-H tool. The experimental results show that the SSO module can effectively improve the query efficiency, reduce the disk I/O cost and make full use of the cluster memory resources.

show abstract

“…1) Multi-Objective Optimization: One approach to achieve multi-objective optimization is to build cost models for each of the heterogeneous computing units. A cost model could be a performance prediction model, also known as surrogate models, to estimate performance efficiency of workload on different platforms for scheduling, e.g., Luo et al [64] presents power-performance characteristic of analytic queries on database systems, and Singhal et al [65], [66], [67] and [68] present similar cost models for relational database, Hive, Hadoop, and Spark platforms deployed on conventional CPU systems. However, none of these include design space for hardware accelerators.…”

Section: Optimization Challengesmentioning

confidence: 99%

Polystore++: Accelerated Polystore System for Heterogeneous Workloads

Singhal

Zhang

Nardi

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

Modern real-time business analytic consist of heterogeneous workloads (e.g., database queries, graph processing, and machine learning). These analytic applications need programming environments that can capture all aspects of the constituent workloads (including data models they work on and movement of data across processing engines). Polystore systems suit such applications; however, these systems currently execute on CPUs and the slowdown of Moore's Law means they cannot meet the performance and efficiency requirements of modern workloads. We envision Polystore++, an architecture to accelerate existing polystore systems using hardware accelerators (e.g., FPGAs, CGRAs, and GPUs). Polystore++ systems can achieve high performance at low power by identifying and offloading components of a polystore system that are amenable to acceleration using specialized hardware. Building a Polystore++ system is challenging and introduces new research problems motivated by the use of hardware accelerators (e.g., optimizing and mapping query plans across heterogeneous computing units and exploiting hardware pipelining and parallelism to improve performance). In this paper, we discuss these challenges in detail and list possible approaches to address these problems.

show abstract

Performance Assurance Model for Applications on SPARK Platform

Cited by 21 publications

References 9 publications

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Query Execution Optimization in Spark SQL

Polystore++: Accelerated Polystore System for Heterogeneous Workloads

Contact Info

Product

Resources

About