Using machine learning to optimize parallelism in big data applications

Abstract

In-memory cluster computing platforms have gained momentum in the last years, due to their ability to analyse big amounts of data in parallel. These platforms are complex and difficult-to-manage environments. In addition, there is a lack of tools to better understand and optimize such platforms that consequently form backbone of big data infrastructure and technologies. This directly leads to underutilization of available resources and application failures in such environment. One of the key aspects that can address this problem is optimization of the task parallelism of application in such environments. In this paper, we propose a machine learning based method that recommends optimal parameters for task parallelization in big data workloads. By monitoring and gathering metrics at system and application level, we are able to find statistical correlations that allow us to characterize and predict the effect of different parallelism settings on performance. These predictions are used to recommend an optimal configuration to users before launching their workloads in the cluster, avoiding possible failures, performance degradation and wastage of resources. We evaluate our method with a benchmark of 15 Spark applications on the Grid5000 testbed. We observe up to a 51% gain on performance when using the recommended parallelism settings. The model is also interpretable and can give insights to the user into how different metrics and parameters affect the performance.

The Deep Learning and Apache Spark Enabled Architecture for Improving the Performance of Big Data Classification

“…Utilizing AI to enhance parallelism in enormous information applications [5] by Hernández proposed an approach utilizing AI to enhance parallelism in huge information. The proposed strategy can precisely anticipate the execution time of a major information based application with various record sizes and parallelism settings utilizing the correct models.…”

“…The conventional knowledge requires the gathering of the considerable number of information over the world to a focal server farm area, to be handled utilizing information parallel applications. [5]. Hence, we need the unique instruments for enormous information.…”

A survey of big data and machine learning

“…A method to address the problem of suggesting the most suitable components for each user by creating a recommender system using intelligent data analysis is proposed in [15]. Reference [16] trains a ML model to predict the duration of big data workloads.…”

GPU-based efficient join algorithms on Hadoop

“…In recent years, many studies have chosen FPGA as an option for hardware accelerators [8,9], including the query [10]. Similarly, image processors (GPU) have been widely used in the field of query processing [11].…”