MapReduce programming and cost-based optimization?

Abstract

MapReduce has emerged as a viable competitor to database systems in big data analytics. MapReduce programs are being written for a wide variety of application domains including business data processing, text analysis, natural language processing, Web graph and social network analysis, and computational science. However, MapReduce systems lack a feature that has been key to the historical success of database systems, namely, cost-based optimization. A major challenge here is that, to the MapReduce system, a program consists of black-box map and reduce functions written in some programming language like C++, Java, Python, or Ruby. Starfish is a self-tuning system for big data analytics that includes, to our knowledge, the first Cost-based Optimizer for simple to arbitrarily complex MapReduce programs. Starfish also includes a Profiler to collect detailed statistical information from unmodified MapReduce programs, and a What-if Engine for fine-grained cost estimation. This demonstration will present the profiling, what-if analysis, and cost-based optimization of MapReduce programs in Starfish. We will show how (nonexpert) users can employ the Starfish Visualizer to (a) get a deep understanding of a MapReduce program's behavior during execution, (b) ask hypothetical questions on how the program's behavior will change when parameter settings, cluster resources, or input data properties change, and (c) ultimately optimize the program.

Towards optimizing the execution of spark scientific workflows using machine learning‐based parameter tuning

“…We obtained 19 papers 20‐22,32,75‐89 . These papers were published in five distinct journals publications, 13 conferences/workshops and one PhD thesis.…”

“…Starfish 21,80,81 is a system that implements a cost‐based optimization approach to help users identify the best parameter values for MapReduce applications. The system consists of three modules: a profiler, which obtains workflow execution statistics and cost estimate; what‐if , which determines the impact of a parameter setting on performance based on the information collected by the profiler; and finally, the optimizer that provides the settings that perform an efficient execution through queries to the what‐if module.…”

“…The approach effectiveness depends on the accuracy of the what‐if module that is based on estimates and in simulations. Also, the cost‐based method implemented by Starfish 21,32,80,81 is based on in‐depth knowledge about the system internal features, and also about the workflow (which may be not available). This challenge has motivated the exploration of optimization methods based on machine learning techniques 20,22 …”

Modeling and optimizing MapReduce programs

“…This is parameter tuning, which is complementary to our approach of a structural program optimization. Similarly, Herodotou et al perform parameter tuning, in this case, based on clustering.…”

Towards the optimization of a parallel streaming engine for telco applications

“…It also attempted to find correlations between these factors in order to better understand how to build a unified and generic performance estimation model. The Starfish system [15, 16] is the most relevant system for our work. It represents a cost‐based optimizer for MapReduce programs which focuses on the optimization of configuration parameters for executing these programs on the Hadoop platform.…”