Adaptive performance model for dynamic scaling Apache Spark Streaming

Petrov, Max; Butakov, Nikolay; Nasonov, Denis; Melnik, Mikhail

doi:10.1016/j.procs.2018.08.243

Cited by 18 publications

(15 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With increasing training data sizes, the models can achieve high prediction accuracy and near-optimal recommendations. However, obtaining training data can be a costly process, as it requires runs [97] Adjusts number of allocated map/reduce slots Cost functions MROnline [78] Supports aggressive and conservative tuning Gray-box Hill Climbing Ant [29] Supports heterogeneous cluster nodes Genetic Algorithm JellyFish [36] Performs dimensionality reduction of search space Model-based Hill Climbing KERMIT [45] Optimizes CPU and memory for workloads Global and local search Stream T-Storm [125] Optimizes number of Worker processes in Storm Traffic-aware Scheduling Das et al [32] Adapts batch size in Spark Streaming Fixed-Point Iteration DRS [43] Adapts # of Workers and parallelism hint per operator Queueing Theory Drizzle [112] Adapts batch size and grouping in Spark Streaming Group and pre-scheduling Petrov et al [96] Decides # of AWS worker nodes and Spark Executors Cost performance model under different settings to avoid under-fitting. The situation becomes worse as workloads can change dynamically, and unseen applications appear.…”

Section: Discussionmentioning

confidence: 99%

“…Last, to achieve higher throughput, Drizzle further implements various optimization approaches both within and across batches. Petrov et al [96] propose a performance model for stream data processing to adaptively assign optimal resources (i.e., number of worker nodes and Executors) to workloads. The framework collects various statistics and system utilization metrics and then uses the models for deciding when and how to scale the current application to maximize throughput.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotou

Chen

2020

ACM Comput. Surv.

View full text Add to dashboard Cite

Big data processing systems (e.g., Hadoop, Spark, Storm) contain a vast number of configuration parameters controlling parallelism, I/O behavior, memory settings, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators grapple with understanding and tuning them to achieve good performance. We investigate existing approaches on parameter tuning for both batch and stream data processing systems and classify them into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We summarize the pros and cons of each approach and raise some open research problems for automatic parameter tuning.

show abstract

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Herodotou

Chen

2020

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…Das et al [31] applied a robust and effective algorithm that adaptively tunes the batch size for promoting the performance of Spark Streaming. Petrov et al [32] proposed a robust and adaptive performance model for Spark Streaming to achieve the goal of allocating resources dynamically and reducing the total cost.…”

Section: Related Workmentioning

confidence: 99%

Resource Configuration Tuning for Stream Data Processing Systems via Bayesian Optimization

et al. 2022

View full text Add to dashboard Cite

Stream data processing systems are becoming increasingly popular in the big data era. Systems such as Apache Flink typically provide a number (e.g., 30) of configuration parameters to flexibly specify the amount of resources (e.g., CPU cores and memory) allocated for tasks. These parameters significantly affect task performance. However, it is hard to manually tune them for optimal performance for an unknown program running on a given cluster. An automatic as well as fast resource configuration tuning approach is therefore desired. To this end, we propose to leverage Bayesian optimization to automatically tune the resource configurations for stream data processing systems. We first select a machine learning model—Random Forest—to construct accurate performance models for a stream data processing program. We subsequently take the Bayesian optimization (BO) algorithm, along with the performance models, to iteratively search the optimal configurations for a stream data processing program. Experimental results show that our approach improves the 99th-percentile tail latency by a factor of 2.62× on average and up to 5.26× overall. Furthermore, our approach improves throughput by a factor of 1.05× on average and up to 1.21× overall.

show abstract

“…; multiple streams join since such cases/scenarios are non-trivial to solve and may consider for future work. Furthermore, The delay of reporting time in case of missed and delayed events could be reduced by identifying the optimal amount of resources to satisfy the required processing delay under specific stream rate change [59]. In integration with other adaptive streaming techniques, other adaptive techniques such as adaptive load shedding for windowed stream joins [13] and dynamic batch sizing [17] can be integrated with our proposed dynamic windowing based on stream rate change for industrial applications.…”

Section: Challengesmentioning

confidence: 99%

Industry 4.0 towards Forestry 4.0: Fire Detection Use Case

Sahal

Alsamhi

Breslin

et al. 2021

Sensors

View full text Add to dashboard Cite

Forestry 4.0 is inspired by the Industry 4.0 concept, which plays a vital role in the next industrial generation revolution. It is ushering in a new era for efficient and sustainable forest management. Environmental sustainability and climate change are related challenges to promote sustainable forest management of natural resources. Internet of Forest Things (IoFT) is an emerging technology that helps manage forest sustainability and protect forest from hazards via distributing smart devices for gathering data stream during monitoring and detecting fire. Stream processing is a well-known research area, and recently, it has gained a further significance due to the emergence of IoFT devices. Distributed stream processing platforms have emerged, e.g., Apache Flink, Storm, and Spark, etc. Querying windowing is the heart of any stream-processing platform which splits infinite data stream into chunks of finite data to execute a query. Dynamic query window-based processing can reduce the reporting time in case of missing and delayed events caused by data drift.In this paper, we present a novel dynamic mechanism to recommend the optimal window size and type based on the dynamic context of IoFT application. In particular, we designed a dynamic window selector for stream queries considering input stream data characteristics, application workload and resource constraints to recommend the optimal stream query window configuration. A research gap on the likelihood of adopting smart IoFT devices in environmental sustainability indicates a lack of empirical studies to pursue forest sustainability, i.e., sustainable forestry applications. So, we focus on forest fire management and detection as a use case of Forestry 4.0, one of the dynamic environmental management challenges, i.e., climate change, to deliver sustainable forestry goals. According to the dynamic window selector’s experimental results, end-to-end latency time for the reported fire alerts has been reduced by dynamical adaptation of window size with IoFT stream rate changes.

show abstract

Adaptive performance model for dynamic scaling Apache Spark Streaming

Cited by 18 publications

References 2 publications

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

A Survey on Automatic Parameter Tuning for Big Data Processing Systems

Resource Configuration Tuning for Stream Data Processing Systems via Bayesian Optimization

Industry 4.0 towards Forestry 4.0: Fire Detection Use Case

Contact Info

Product

Resources

About