Spark deployment and performance evaluation on the MareNostrum supercomputer

Tous, Rubén; Gounaris, Anastasios; Tripiana, Carlos; Torres, Jordi; Girona, S.; Ayguadé, Eduard; Labarta, Jesús; Becerra, Yolanda; Carrera, David; Valero, Mateo

doi:10.1109/bigdata.2015.7363768

Cited by 22 publications

(42 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we experiment with the MareNostrum petascale supercomputer at the Barcelona Supercomputing Center in Spain. After configuring the cluster in an application-independent way according to the results in [8], we examine the impact of configurable parameters on a range of applications and derive a simple trial-and-error tuning methodology that can be applied to each Spark application separately. We test our methodology using three case studies with particularly encouraging results.…”

Section: Introductionmentioning

confidence: 99%

Spark Parameter Tuning via Trial-and-Error

Petridis

Gounaris

Torres

2016

Advances in Intelligent Systems and Computing

Self Cite

View full text Add to dashboard Cite

Abstract. Spark has been established as an attractive platform for big data analysis, since it manages to hide most of the complexities related to parallelism, fault tolerance and cluster setting from developers. However, this comes at the expense of having over 150 configurable parameters, the impact of which cannot be exhaustively examined due to the exponential amount of their combinations. The default values allow developers to quickly deploy their applications but leave the question as to whether performance can be improved open. In this work, we investigate the impact of the most important of the tunable Spark parameters on the application performance and guide developers on how to proceed to changes to the default values. We conduct a series of experiments with known benchmarks on the MareNostrum petascale supercomputer to test the performance sensitivity. More importantly, we offer a trialand-error methodology for tuning parameters in arbitrary applications based on evidence from a very small number of experimental runs. We test our methodology in three case studies, where we manage to achieve speedups of more than 10 times.

show abstract

Section: Introductionmentioning

confidence: 99%

Spark Parameter Tuning via Trial-and-Error

Petridis

Gounaris

Torres

2016

Advances in Intelligent Systems and Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…5 The profiles for these five steps and the potential repartitioning are shown in Figure 7 when the flow is executed on a cluster from 4 to 16 nodes on MN3.…”

Section: Real Case-studymentioning

confidence: 99%

“…To avoid imbalanced execution, the degree of partitioning must be equal to the number of cores multiplied by a small integer. However, based on (i) the evidence in [5] that, for CPU-intensive applications on MN3, the most efficient configuration of the degree of partitioning is to be set equal to the number of cores, and (ii) the evidence in [6], where the main performance bottlenecks are the CPU ones, in this work, we always set the degree of partitioning to the number of cores. Also, we allow the usage of complete machines, each consisting of 16 cores.…”

Section: Our Setting and The Benchmarking Applicationsmentioning

confidence: 99%

“…At June 2013, MareNostrum was positioned at the 29th place in the TOP500 list of fastest supercomputers in the world, whereas according to the latest TOP500 list in November 2015, MareNostrum is 93rd. A full technical description of MN3 and how it supports Spark applications is in [5]. Spark allows for several cluster managers: standalone, YARN and MESOS.…”

Section: Our Setting and The Benchmarking Applicationsmentioning

confidence: 99%

“…The work in [5] has started to scratch the surface of this issue and, after extensive experimentation in a high-performance computing (HPC) platform, namely the Marenostrum III (MN3), the supercomputer at Barcelona Supercomputing Centre (BSC), and an additional commer-…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamic Configuration of Partitioning in Spark Applications

Gounaris

Kougka

Tous

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

Abstract-Spark has become one of the main options for large-scale analytics running on top of shared-nothing clusters. This work aims to make a deep dive into the parallelism configuration and shed light on the behavior of parallel spark jobs. It is motivated by the fact that running a Spark application on all the available processors does not necessarily imply lower running time, while may entail waste of resources. We first propose analytical models for expressing the running time as a function of the number of machines employed. We then take another step, namely to present novel algorithms for configuring dynamic partitioning with a view to minimizing resource consumption without sacrificing running time beyond a user-defined limit. The problem we target is NP-hard. To tackle it, we propose a greedy approach after introducing the notions of dependency graphs and of the benefit from modifying the degree of partitioning at a stage; complementarily, we investigate a randomized approach. Our polynomial solutions are capable of judiciously use the resources that are potentially at user's disposal and strike interesting trade-offs between running time and resource consumption. Their efficiency is thoroughly investigated through experiments based on real execution data.

show abstract

On the Performance of Spark on HPC Systems: Towards a Complete Picture

Yildiz

Ibrahim

2018

Supercomputing Frontiers

View full text Add to dashboard Cite

Big Data analytics frameworks (e.g., Apache Hadoop and Apache Spark) have been increasingly used by many companies and research labs to facilitate large-scale data analysis. However, with the growing needs of users and size of data, commodity-based infrastructure will strain under the heavy weight of Big Data. On the other hand, HPC systems offer a rich set of opportunities for Big Data processing. As first steps toward Big Data processing on HPC systems, several research efforts have been devoted to understanding the performance of Big Data applications on these systems. Yet the HPC specific performance considerations have not been fully investigated. In this work, we conduct an experimental campaign to provide a clearer understanding of the performance of Spark, the de facto in-memory data processing framework, on HPC systems. We ran Spark using representative Big Data workloads on Grid'5000 testbed to evaluate how the latency, contention and file system's configuration can influence the application performance. We discuss the implications of our findings and draw attention to new ways (e.g., burst buffers) to improve the performance of Spark on HPC systems.

show abstract

Spark deployment and performance evaluation on the MareNostrum supercomputer

Cited by 22 publications

References 12 publications

Spark Parameter Tuning via Trial-and-Error

Spark Parameter Tuning via Trial-and-Error

Dynamic Configuration of Partitioning in Spark Applications

On the Performance of Spark on HPC Systems: Towards a Complete Picture

Contact Info

Product

Resources

About