Spark Versus Flink: Understanding Performance in Big Data Analytics Frameworks

Abstract

Abstract-Big Data analytics has recently gained increasing popularity as a tool to process large amounts of data on-demand. Spark and Flink are two Apache-hosted data analytics frameworks that facilitate the development of multi-step data pipelines using directly acyclic graph patterns. Making the most out of these frameworks is challenging because efficient executions strongly rely on complex parameter configurations and on an in-depth understanding of the underlying architectural choices. Although extensive research has been devoted to improving and evaluating the performance of such analytics frameworks, most of them benchmark the platforms against Hadoop, as a baseline, a rather unfair comparison considering the fundamentally different design principles. This paper aims to bring some justice in this respect, by directly evaluating the performance of Spark and Flink. Our goal is to identify and explain the impact of the different architectural choices and the parameter configurations on the perceived end-to-end performance. To this end, we develop a methodology for correlating the parameter settings and the operators execution plan with the resource usage. We use this methodology to dissect the performance of Spark and Flink with several representative batch and iterative workloads on up to 100 nodes. Our key finding is that there none of the two framework outperforms the other for all data types, sizes and job patterns. This paper performs a fine characterization of the cases when each framework is superior, and we highlight how this performance correlates to operators, to resource usage and to the specifics of the internal framework design.

Benchmarking Distributed Stream Data Processing Systems

“…The Big-DataBench [8] suite contains 19 scenarios covering a broad range of applications and diverse data sets. Marcu et al [9] performed an extensive analysis of the differences between Apache Spark and Apache Flink on iterative workloads. The above benchmarks either adopt batch processing systems and metrics used in batch processing systems or apply the batchbased metrics on SDPSs.…”

Microservice-Based Platform for Space Situational Awareness Data Analytics

“…Apache Hadoop has been used in various big data processing fields but cannot meet the real-time computing tasks and requirements [44,45]. Apache Storm only supports stream processing [46], Apache Spark simulates stream processing based on batch processing [47], and Apache Flink is entirely based on stream processing and simulates batch processing through stream processing [48]. Apache Flink can implement both stream processing and batch processing via a single solution, which can help prevent duplication of codes during development.…”

An experimental survey on big data frameworks

“…We mention that most of the above presented surveys are limited in terms of both the evaluated features of Big Data frameworks and the number of considered frameworks. For example, in [64], only stream processing frameworks are considered while in [16] [54] [24] [40], only batch processing frameworks are considered. We highlight that our experimental survey differs from the above presented works by the fact that it compares the studied frameworks in the case of both batch and stream processing.…”