Hadoop on HPC: Integrating Hadoop and Pilot-Based Dynamic Resource Management

Abstract

Abstract-High-performance computing platforms such as "supercomputers" have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as net producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the traditional limitations of HPC platforms. There exist a class of scientific applications however, that need the collective capabilities of traditional high-performance computing environments and the Apache Hadoop ecosystem. For example, the scientific domains of bio-molecular dynamics, genomics and network science need to couple traditional computing with Hadoop/Spark based analysis. We investigate the critical question of how to present the capabilities of both computing environments to such scientific applications. Whereas this questions needs answers at multiple levels, we focus on the design of resource management middleware that might support the needs of both. We propose extensions to the Pilot-Abstraction so as to provide a unifying resource management layer. This is an important step towards interoperable use of HPC and Hadoop/Spark. It also allows applications to integrate HPC stages (e. g. simulations) to data analytics. Many supercomputing centers have started to officially support Hadoop environments, either in a dedicated environment or in hybrid deployments using tools such as myHadoop. This typically involves many intrinsic, environment-specific details that need to be mastered, and often swamp conceptual issues like: How best to couple HPC and Hadoop application stages? How to explore runtime trade-offs (data localities vs. data movement)? This paper provides both conceptual understanding and practical solutions to the integrated use of HPC and Hadoop environments. Our experiments are performed on state-of-the-art production HPC environments and provide middleware for multiple domain sciences.

Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications

“…While some ensemble applications are data-flow oriented and thus amenable to be implemented with MapReduce, EnTK adopts a more flexible and coarse-grained notion of tasks, where a task in EnTK can support multiple programming models, including MPI. Further, EnTK does not assume a specific runtime system and, in conjunction with RP, can use Hadoop on HPC [21].…”

Task-parallel Analysis of Molecular Dynamics Trajectories

“…TACC Wrangler has 24 Haswell hyper-threading enabled cores/node and 128 GB memory/node (120 nodes). Experiments were carried using RADICAL-Pilot and Pilot-Spark [19] extension, which allows to efficiently manage Spark on HPC resources through a common resource management API. We utilize a set of custom scripts to start the Dask cluster.…”

RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms

“…That integration allows users to create Hadoop jobs with ease via a GUI but it also requires users to setup and configure a distributed Hadoop cluster on the HPC platform. Hadoop-on-HPC [2] integrates Hadoop with RADICAL-Pilot (RP): Hadoop extends RP with the MapReduce programming paradigm, and RP's pilot capabilities enable deployment of the Hadoop cluster and scheduling of the workflow's tasks on that cluster.…”

“…We integrated two existing, independently developed software systems and executed the workflows of two exemplar use cases on HPC resources. Based on our previous experience with integrating independent systems [1], [2], [11] and our building blocks approach [3], we describe an integration that minimizes changes in the existing code-bases while allowing users to benefit from the capabilities of both systems. We show how to approach the integration, evaluate diverse integration points and align the programming and execution models of the two systems.…”