Analytical composite performance models for Big Data applications

Karimian-Aliabadi, Soroush; Ardagna, Danilo; Entezari‐Maleki, Reza; Gianniti, Eugenio; Movaghar, Ali

doi:10.1016/j.jnca.2019.06.009

Cited by 16 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [43], the lumping technique was used to alleviate the scalability issue of predicting YARN environment, introducing a significant error of about 15%. The contribution was built upon the efforts for developing analytical models for single queue scenarios in [44], [45]. Our findings from related work are summarized in Table 9.…”

Section: Related Workmentioning

confidence: 99%

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Karimian-Aliabadi

Aseman-Manzar

Entezari‐Maleki

et al. 2023

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

Companies depend on mining data to grow their business more than ever. To achieve optimal performance of Big Data analytics workloads, a careful configuration of the cluster and the employed software framework is required. The lack of flexible and accurate performance models, however, render this a challenging task. This paper fills this gap by presenting accurate performance prediction models based on Stochastic Activity Networks (SANs). In contrast to existing work, the presented models consider multiple work queues, a critical feature to achieve high accuracy in realistic usage scenarios. We first introduce a monolithic analytical model for a multi-queue YARN cluster running DAG-based Big Data applications that models each queue individually. To overcome the limited scalability of the monolithic model, we then present a fixed-point model that iteratively computes the throughput of a single queue with respect to the rest of the system until a fixed-point is reached. The models are evaluated on a real-world cluster running the widely-used Apache Spark framework and the YARN scheduler. Experiments with the common transaction-based TPC-DS benchmark show that the proposed models achieve an average error of only 5.6% in predicting the execution time of the Spark jobs. The presented models enable businesses to optimize their cluster configuration for a given workload and thus to reduce their expenses and minimize service level agreement (SLA) violations. Makespan minimization and per-stage analysis are examined as representative efforts to further assess the applicability of our proposition.

show abstract

Section: Related Workmentioning

confidence: 99%

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Karimian-Aliabadi

Aseman-Manzar

Entezari‐Maleki

et al. 2023

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The "Action Program for Promoting the Development of Big Data" clearly supports companies in developing third-party data analysis and discovery services, technology outsourcing services, and knowledge process outsourcing services based on big data, and encourages large and medium-sized enterprises to coordinate development and collaborative innovation. Therefore, SMEs can adopt flexible and diverse forms such as third-party leasing and purchasing professional services or using Cloud deployement creatively to achieve comprehensive coverage and improvement of big data infrastructure [6].…”

Section: Increase Investment and Improve Big Data Infrastructurementioning

confidence: 99%

The Difficulties and Optimization Mechanism of Big Data Application in SMEs

Liu¹,

LI²

2022

Advances in Social Science, Education and Humanities Research

View full text Add to dashboard Cite

In the era of digital economy, big data has become a strategic resource, and big data application capabilities have also become a strategic capability, which can effectively assist SMEs in precision marketing, enhancing corporate competitiveness, and achieving quality and efficiency improvements. However, the application of big data for SMEs is facing a series of difficulties such as insufficient understanding and lack of big data thinking, not enough investment and incomplete big data infrastructure, irregular management and low quality of basic big data, incomplete introduction and lack of big data talents. For this reason, it is necessary to raise awareness and establish big data thinking, increase investment and improve big data infrastructure, standardize management and improve the quality of basic big data, and pay equal attention to talent introduction and education, and strengthen the construction of big data talents.

show abstract

“…Castiglione et al [6] focus on predicting metrics relevant for operating highly concurrent applications in cloud infrastructure such as performance, number of virtual machines or energy efficiency. Aliabadi et al [2] focus on predicting the performance of batch applications in different Big Data analysis frameworks using Stochastic Activity Networks. These approaches for batch processing do not support the metrics required for stream processing, which we focus on.…”

Section: Related Workmentioning

confidence: 99%

Accurate Performance Predictions with Component-Based Models of Data Streaming Applications

Werle

Seifermann

Koziolek

2022

Software Architecture

View full text Add to dashboard Cite

Data streaming applications are an important class of dataintensive systems and performance is an essential quality of such systems. Current component-based performance prediction approaches are not sufficient for modeling and predicting the performance of those systems, because the models require elaborate manual engineering to approximate the behavior of data streaming applications that include stateful asynchronous operations, such as windowing operations, and because the simulations for these models do not support the metrics that are specific to data streaming applications. In this paper, we present a modeling language, a simulation and a case-study-based evaluation of the prediction accuracy of an approach for modeling systems that contain stateful asynchronous operations. Our approach directly represents these operations and simulates their behavior. We compare measurements of relevant performance metrics to performance simulation results for a system that processes smart meter readings. To assess the prediction accuracy of our model, we vary both the configuration of the streaming application, such as window sizes, as well as the characteristics of the input data, i.e., the number of smart meters. Our evaluation shows that our model yields prediction results that are competitive with a state-of-the-art baseline model without incurring the additional manual engineering overhead.

show abstract

Analytical composite performance models for Big Data applications

Cited by 16 publications

References 38 publications

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

Fixed-Point Iteration Approach to Spark Scalable Performance Modeling and Evaluation

The Difficulties and Optimization Mechanism of Big Data Application in SMEs

Accurate Performance Predictions with Component-Based Models of Data Streaming Applications

Contact Info

Product

Resources

About