Introducing map-reduce to high end computing

Abstract

In this work we present an scientific application that has been given a Hadoop MapReduce implementation. We also discuss other scientific fields of supercomputing that could benefit from a MapReduce implementation. We recognize in this work that Hadoop has potential benefit for more applications than simply datamining, but that it is not a panacea for all data intensive applications.We provide an example of how the halo finding application, when applied to large astrophysics datasets, benefits from the model of the Hadoop architecture. The halo finding application uses a friends of friends algorithm to quickly cluster together large sets of particles to output files which a visualization software can interpret. The current implementation requires that large datasets be moved from storage to computation resources for every simulation of astronomy data. Our Hadoop implementation allows for an in-place halo finding application on the datasets, which removes the time consuming process of tranferring data between resources.

Availability Of JobTracker Machine In Hadoop/MapReduce Zookeeper Coordinated Clusters

“…ZooKeeper is a high-performance, available and scalable open-source software package [6] that is used to coordinate distributed systems. It is a stripped-down file system that exposes some primitives on which distributed systems can build to implement higher level services such as synchronization, naming, and configuration management.…”

SecureMR: A Service Integrity Assurance Framework for MapReduce

“…Ekanayake et al [4] applied MapReduce technique for two scientific analyses, High Energy Physics data analyses and Kmeans clustering. Mackey et al [3] utilized MapReduce for High End Computing applications. Most of them focus on how to utilize MapReduce to solve issues or problems in specific application domains.…”

“…Recently, many organizations have adopted the model of MapReduce, and developed their own implementations of MapReduce, such as Google MapReduce [1] and Yahoo's Hadoop [2], as well as thousands of MapReduce applications. Moreover, MapReduce has been adopted by many academic researchers for data processing in different research areas, such as high end computing [3], data intensive scientific analysis [4], large scale semantic annotation [5] and machine learning [6].…”

Efficient Traffic Management with Intermediate Data Partition for Big Data Applications

“…Map Reduce [1] [2] [3] implementation with Hadoop [4] [5] has been implemented by most MNC's organisations and companies, ie, Yahoo!, Google and Facebook, for various big data applications, like machine learning [6] [7] [8], bioinformatics [9] [10] [11], and cyber security [12] [13]. Computation phase mainly divides a data into phases, namely first phase which map and the second is reduce phase, which in turn are computed with several map tasks and then the secondary phase reduce tasks.…”