Anatomy of machine learning algorithm implementations in MPI, Spark, and Flink

Abstract

With the ever-increasing need to analyze large amounts of data to get useful insights, it is essential to develop complex parallel machine learning algorithms that can scale with data and number of parallel processes. These algorithms need to run on large data sets as well as they need to be executed with minimal time in order to extract useful information in a time-constrained environment. Message passing interface (MPI) is a widely used model for developing such algorithms in high-performance computing paradigm, while Apache Spark and Apache Flink are emerging as big data platforms for large-scale parallel machine learning. Even though these big data frameworks are designed differently, they follow the data flow model for execution and user APIs. Data flow model offers fundamentally different capabilities than the MPI execution model, but the same type of parallelism can be used in applications developed in both models. This article presents three distinct machine learning algorithms implemented in MPI, Spark, and Flink and compares their performance and identifies strengths and weaknesses in each platform.

Twister2: Design of a big data toolkit

“…With our previous work 9 we have observed that various decisions made at different components of a big data runtime determine the type of applications that can be executed efficiently. The layered architecture proposed in this work will eliminate the monolithic designs and empower components to be developed independently and efficiently.…”

“…AMT systems mostly focus on computationally intensive applications, and there is ongoing research to make them more efficient and productive. We find that big data systems developed according to a dataflow model are inefficient in computationally intensive applications with tightly synchronized parallel operations 9 , while AMT systems are not optimized for data processing.…”

A Survey on Vertical and Horizontal Scaling Platforms for Big Data Analytics

“…Flink uses thread-based worker model for executing the data flow graphs. It can chain consecutive tasks in the workflow in a single node to make the run more efficient by reducing data serializations and communications [37]. Flink and Spark are designed to make Hadoop scalable and fault-tolerant, and to analyze intensive data applications with distributed memory framework.…”

Task-parallel Analysis of Molecular Dynamics Trajectories

“…Frameworks for parallel data analysis have been created by the High Performance Computing (HPC) and Big Data communities [17]. MPI is the most used programming model for HPC resources.…”

“…In addition, the MapReduce [7] abstraction makes it easy to exploit data-parallelism as required by many analysis applications. Several recent publications applied HPC techniques to advance traditional Big Data applications and Big Data frameworks [17].…”