Benchmarking MapReduce Implementations for Application Usage Scenarios

Fadika, Zacharia; Dede, Elif; Govindaraju, Madhusudhan; Ramakrishnan, Lavanya

doi:10.1109/grid.2011.21

Cited by 22 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2011 Fadika et al [9] presented a performance evaluation study to compare MapReduce platforms under a wide range of use cases. They compared the performance of MapReduce, Apache Hadoop, Twister, and LEMO.…”

Section: Related Workmentioning

confidence: 99%

Performance Analysis in Bigdata

Kaur¹,

Kaur²,

Kaur³

2015

IJITCS

View full text Add to dashboard Cite

Abstract-Big data technologies like Hadoop, NoSQL, Messaging Queues etc. helps in BigData analytics, drive business growth and to take right decisions in time. These Big Data environments are very dynamic and complex; they require performance validation, root cause analysis, and tuning to ensure success. In this paper we talk about how we can analyse and test the performance of these systems. We present the important factors in a big data that are primary candidates for performance testing like data ingestion capacity and throughput, data processing capacity, simulation of expected usage, map reduce jobs and so on and suggest measures to improve performance of bigdata Index Terms-NOSQL, YCSB, HDFS.

show abstract

Section: Related Workmentioning

confidence: 99%

Performance Analysis in Bigdata

Kaur¹,

Kaur²,

Kaur³

2015

IJITCS

View full text Add to dashboard Cite

show abstract

“…Magellan staff also worked closely with the Grid Computing Research Laboratory at SUNY, Binghamton in a comparative benchmarking study of MapReduce implementations and an alternate implementation of MapReduce that can work in HPC environments. This collaboration resulted in two publications in Grid 2011 [25,24].…”

Section: Collaborations and Synergistic Activitiesmentioning

confidence: 99%

“…In addition Magellan staff participated in the study. The paper was published in Grid 2011 [24] In this study, we compare and study the performance of Hadoop [4], Twister [22] and LEMO-MR [26] in various real-world application usage scenarios, including data-intensive, iterative, CPU-intensive, and memory-intensive loads. We not only compare the chosen frameworks in several real-world application scenarios, but also in real-world cluster scenarios, including fault-prone clusters/applications, physically heterogeneous clusters and load-imbalanced clusters.…”

Section: Comparison Of Mapreduce Implementationsmentioning

confidence: 99%

The Magellan Final Report on Cloud Computing

Coghlan¹

2011

View full text Add to dashboard Cite

Executive SummaryThe goal of Magellan, a project funded through the U.S. Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR), was to investigate the potential role of cloud computing in addressing the computing needs for the DOE Office of Science (SC), particularly related to serving the needs of midrange computing and future data-intensive computing workloads. A set of research questions was formed to probe various aspects of cloud computing from performance, usability, and cost. To address these questions, a distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Center (NERSC). The testbed was designed to be flexible and capable enough to explore a variety of computing models and hardware design points in order to understand the impact for various scientific applications. During the project, the testbed also served as a valuable resource to application scientists. Applications from a diverse set of projects such as MG-RAST (a metagenomics analysis server), the Joint Genome Institute, the STAR experiment at the Relativistic Heavy Ion Collider, and the Laser Interferometer Gravitational Wave Observatory (LIGO), were used by the Magellan project for benchmarking within the cloud, but the project teams were also able to accomplish important production science utilizing the Magellan cloud resources.Cloud computing has garnered significant attention from both industry and research scientists as it has emerged as a potential model to address a broad array of computing needs and requirements such as custom software environments and increased utilization among others. Cloud services, both private and public, have demonstrated the ability to provide a scalable set of services that can be easily and cost-effectively utilized to tackle various enterprise and web workloads. These benefits are a direct result of the definition of cloud computing: on-demand self-service resources that are pooled, can be accessed via a network, and can be elastically adjusted by the user. The pooling of resources across a large user base enables economies of scale, while the ability to easily provision and elastically expand the resources provides flexible capabilities.Following the Executive Summary we summarize the key findings and recommendations of the project. Greater detail is provided in the body of the report. Here we briefly summarize some of the high-level findings from the study.• Cloud approaches provide many advantages, including customized environments that enable users to bring their own software stack and try out new computing environments without significant administration overhead, the ability to quickly surge resources to address larger problems, and the advantages that come from increased economies of scale. Virtualization is the primary strategy of providing these capabilities. Our experience working with application scientists using the cloud demonstrated the power of virtualization to enable ...

show abstract

“…CPU-intensive applications tend to utilize CPU 100% but CPU-light applications do not [16]. Memory usage, disk usage, network traffic usage etc.…”

Section: Realization Of Testsmentioning

confidence: 99%

“…Fadika et al performed tests to realize which framework shows high operating performance on which works by comparing two different MapReduce frameworks with Hadoop. In their study, tested applications are classified as CPU-intensive, memory-intensive, or data-intensive applications [16].…”

Section: Introductionmentioning

confidence: 99%

Hadoop framework implementation and performance analysis on a cloud

Özen¹,

Tekerek²,

Sultanov³

2017

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

Abstract:The Hadoop framework uses the MapReduce programming paradigm to process big data by distributing data across a cluster and aggregating. MapReduce is one of the methods used to process big data hosted on large clusters. In this method, jobs are processed by dividing into small pieces and distributing over nodes. Parameters such as distributing method over nodes, the number of jobs held in a parallel fashion, and the number of nodes in the cluster affect the execution time of jobs. The aim of this paper is to determine how the numbers of nodes, maps, and reduces affect the performance of the Hadoop framework in a cloud environment. For this purpose, tests were carried out on a Hadoop cluster with 10 nodes hosted in a cloud environment by running PiEstimator, Grep, Teragen, and Terasort benchmarking tools on it. These benchmarking tools available under the Hadoop framework are classified as CPU-intensive and CPU-light applications as a result of tests. In CPU-light applications, increasing the numbers of nodes, maps, and reduces does not improve the efficiency of these applications; they even cause an increase in time spent on jobs by using system resources unnecessarily. Therefore, in CPU-light applications, selecting the numbers of nodes, maps, and reduces as minimum is found as the optimization of time spent on a process. In CPU-intensive applications, according to the phase that small job pieces is processed, it is found that selecting the number of maps or reduces equal to total number of CPUs on a cluster is the optimization of time spent on a process.

show abstract

Benchmarking MapReduce Implementations for Application Usage Scenarios

Cited by 22 publications

References 17 publications

Performance Analysis in Bigdata

Performance Analysis in Bigdata

The Magellan Final Report on Cloud Computing

Hadoop framework implementation and performance analysis on a cloud

Contact Info

Product

Resources

About