BSP cost and scalability analysis for MapReduce operations

Abstract

Data abundance poses the need for powerful and easy-to-use tools that support processing large amounts of data. MapReduce has been increasingly adopted for over a decade by many companies, and more recently, it has attracted the attention of an increasing number of researchers in several areas. One main advantage is that the complex details of parallel processing, such as complex network programming, task scheduling, data placement, and fault tolerance, are hidden in a conceptually simple framework. MapReduce is supported by mature software technologies for deployment in data centers such as Hadoop. As MapReduce becomes popular for high-performance applications, many questions arise concerning its performance and efficiency.In this paper, we demonstrated formally lower bounds on the isoefficiency function for MapReduce applications, when these applications can be modeled as BSP jobs. We also demonstrate how communication and synchronization costs can be dominant for MapReduce computations and discuss the conditions under which such scalability limits are valid. To our knowledge, this is the first study that demonstrates scalability bounds for MapReduce applications. We also discuss how some MapReduce implementations such as Hadoop can mitigate such costs to approach linear, or near-to-linear speedups.

SparkBLAST: scalable BLAST processing using in-memory operations

“…This is the most scalable configuration over all scenarios analyzed in [5]. Simulation results that goes up to 10000 nodes corroborate the limits stated in this and other theorems of [5]. …”

“…In this scenario, MapReduce (and its Hadoop implementation) emerged as a paramount framework that supports design patterns which represent general reusable solutions to commonly occurring problems across a variety of problem domains including analysis and assembly of biological sequences [ 4 ]. MapReduce has delivered outstanding performance and scalability for a myriad of applications running over hundreds to thousands of processing nodes [ 5 ]. On the other hand, over the last decade, cloud computing has emerged as a powerful platform for the agile and dynamic provisioning of computational resources for computational and data intensive problems.…”

A Systematic Review of Healthcare Big Data

“…MapReduce is a programming framework that enables us to process massive amount of data in parallel in a distributed computing environment. is framework consists of two main functions, namely, Map and Reduce that can effectively manage structured as well as unstructured data [95,96]. As the name MapReduce indicates, reducer function occurs after the completion of the mapper function.…”

Parallel and distributed computing for Big Data applications

“…In , the scalability of MapReduce computations is studied. Proposed in 2004, MapReduce became very popular and widespread, being adopted for the execution of various Big Data applications such as web indexing, social network applications, processing logs and raw data, queries processing on large‐volume datasets, and many others.…”