MARIANE: MApReduce Implementation Adapted for HPC Environments

Fadika, Zacharia; Dede, Elif; Govindaraju, Madhusudhan; Ramakrishnan, Lavanya

doi:10.1109/grid.2011.20

Cited by 28 publications

(37 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detailed performance results are available in the paper [25]. With this framework, we eliminate the need for an additional file system and along with it, the involvement of MapReduce in expensive file system maintenance operations.…”

Section: Discussionmentioning

confidence: 98%

“…Magellan staff also worked closely with the Grid Computing Research Laboratory at SUNY, Binghamton in a comparative benchmarking study of MapReduce implementations and an alternate implementation of MapReduce that can work in HPC environments. This collaboration resulted in two publications in Grid 2011 [25,24].…”

Section: Collaborations and Synergistic Activitiesmentioning

confidence: 99%

“…Recently, there have been a number of implementations of MapReduce and similar data processing tools [21,25,36,45,59,73,87]. Apache Hadoop was the most popular implementation of MapReduce at the start of the Magellan project and it continuous to gain traction in various communities.…”

Section: Mapreduce Programming Modelmentioning

confidence: 99%

“…In addition Magellan staff participated in the study. The paper was published in Grid 2011 [25] The MapReduce model in its most popular form, Hadoop [4], uses the Hadoop Distributed File System (HDFS) [76] to serve as the Input/Output manager and fault-tolerance support system for the framework. The use of Hadoop, and of the HDFS is however not directly compatible with HPC environments.…”

Section: Marianementioning

confidence: 99%

See 3 more Smart Citations

The Magellan Final Report on Cloud Computing

Coghlan¹

2011

View full text Add to dashboard Cite

Executive SummaryThe goal of Magellan, a project funded through the U.S. Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR), was to investigate the potential role of cloud computing in addressing the computing needs for the DOE Office of Science (SC), particularly related to serving the needs of midrange computing and future data-intensive computing workloads. A set of research questions was formed to probe various aspects of cloud computing from performance, usability, and cost. To address these questions, a distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Center (NERSC). The testbed was designed to be flexible and capable enough to explore a variety of computing models and hardware design points in order to understand the impact for various scientific applications. During the project, the testbed also served as a valuable resource to application scientists. Applications from a diverse set of projects such as MG-RAST (a metagenomics analysis server), the Joint Genome Institute, the STAR experiment at the Relativistic Heavy Ion Collider, and the Laser Interferometer Gravitational Wave Observatory (LIGO), were used by the Magellan project for benchmarking within the cloud, but the project teams were also able to accomplish important production science utilizing the Magellan cloud resources.Cloud computing has garnered significant attention from both industry and research scientists as it has emerged as a potential model to address a broad array of computing needs and requirements such as custom software environments and increased utilization among others. Cloud services, both private and public, have demonstrated the ability to provide a scalable set of services that can be easily and cost-effectively utilized to tackle various enterprise and web workloads. These benefits are a direct result of the definition of cloud computing: on-demand self-service resources that are pooled, can be accessed via a network, and can be elastically adjusted by the user. The pooling of resources across a large user base enables economies of scale, while the ability to easily provision and elastically expand the resources provides flexible capabilities.Following the Executive Summary we summarize the key findings and recommendations of the project. Greater detail is provided in the body of the report. Here we briefly summarize some of the high-level findings from the study.• Cloud approaches provide many advantages, including customized environments that enable users to bring their own software stack and try out new computing environments without significant administration overhead, the ability to quickly surge resources to address larger problems, and the advantages that come from increased economies of scale. Virtualization is the primary strategy of providing these capabilities. Our experience working with application scientists using the cloud demonstrated the power of virtualization to enable ...

show abstract

Section: Discussionmentioning

confidence: 98%

Section: Collaborations and Synergistic Activitiesmentioning

confidence: 99%

Section: Mapreduce Programming Modelmentioning

confidence: 99%

Section: Marianementioning

confidence: 99%

See 2 more Smart Citations

The Magellan Final Report on Cloud Computing

Coghlan¹

2011

View full text Add to dashboard Cite

show abstract

“…Furthermore, open sourcing Hadoop [3] implementation of MapReduce has resulted in a wide adoption of this paradigm. Although, Hadoop remains the most popular implementation of MapReduce, many other implementations have been introduced such as Disco [11], and MARIANE [12].…”

Section: A Mapreduce Based Systemsmentioning

confidence: 99%

A Scalability Comparison Study of Data Management Approaches for Smart Metering Systems

Chihoub

Collet

2016

2016 45th International Conference on Parallel Processing (ICPP)

View full text Add to dashboard Cite

Abstract-Nowadays, more and more data are being generated and collected in electrical smart grids. Most of these data are coming from smart meters and sensors deployed massively throughout the power grid. As the generation of data is becoming ever more frequent and with the constantly increasing volumes, it is becoming harder and harder to manage and process these data at the scale of a smart grid within legacy systems. In this work, we focus on investigating the scalability and performance of different data management approaches for meter data processing. To this end, we conduct a thorough experimental study of various systems including a parallel relational database system, MapReduce based systems including Hadoop and Spark, and a NoSQL datastore system. Our experiment sets were conducted on up to 140 nodes on Grid5000 and up to 1.4 TB of meter data. Our results demonstrate that parallel relational systems are more suited for most processing types on smart meter data in the smart grid. In contrast, we show that with the appropriate distribution model, data partitioning and modeling choices we achieve very fast and scalable bill computations, the main complex processing for utilities providers.

show abstract

Experiences of Converging Big Data Analytics Frameworks with High Performance Computing Systems

Cheng

et al. 2018

Supercomputing Frontiers

View full text Add to dashboard Cite

Abstract. With the rapid development of big data analytics frameworks, many existing high performance computing (HPC) facilities are evolving new capabilities to support big data analytics workloads. However, due to the different workload characteristics and optimization objectives of system architectures, migrating data-intensive applications to HPC systems that are geared for traditional compute-intensive applications presents a new challenge. In this paper, we address a critical question on how to accelerate complex application that contains both data-intensive and compute-intensive workloads on the Tianhe-2 system by deploying an in-memory file system as data access middleware; we characterize the impact of storage architecture on data-intensive MapReduce workloads when using Lustre as the underlying file system. Based on our characterization and findings of the performance behaviors, we propose shared map output shuffle strategy and file metadata cache layer to alleviate the impact of metadata bottleneck. The evaluation of these optimization techniques shows up to 17% performance benefit for data-intensive workloads.

show abstract

MARIANE: MApReduce Implementation Adapted for HPC Environments

Cited by 28 publications

References 11 publications

The Magellan Final Report on Cloud Computing

The Magellan Final Report on Cloud Computing

A Scalability Comparison Study of Data Management Approaches for Smart Metering Systems

Experiences of Converging Big Data Analytics Frameworks with High Performance Computing Systems

Contact Info

Product

Resources

About