CATCH: A Cloud-Based Adaptive Data Transfer Service for HPC

Abstract

Abstract-Modern High Performance Computing (HPC)applications process very large amounts of data. A critical research challenge lies in transporting input data to the HPC center from a number of distributed sources, e.g., scientific experiments and web repositories, etc., and offloading the result data to geographically distributed, intermittently available endusers, often over under-provisioned connections. Such enduser data services are typically performed using point-to-point transfers that are designed for well-endowed sites and are unable to reconcile the center's resource usage and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. To overcome these inefficiencies, decentralized HPC data services are emerging as viable alternatives. In this paper, we develop and enhance such distributed data services by designing CATCH, a Cloud-based Adaptive data Transfer serviCe for HPC. CATCH leverages a bevy of cloud storage resources to orchestrate a decentralized data transport with fail-over capabilities. Our results demonstrate that CATCH is a feasible approach, and can help improve the data transfer times at the HPC center by as much as 81.1% for typical HPC workloads.

Cloud-aware data intensive workflow scheduling on volunteer computing systems

“…ElasticSite [39] sent a part of the Grid workload to the Cloud when there is an overload of the user demand. CATCH [37] utilized the Cloud storage service for better data access between the desktop worker and the HPC center. Our work focuses on using the VC resources and only the missed deadline tasks are re-scheduled on the Cloud resources to improve the percentage of successful workflow.…”

“…CyberShake [27] that is used by the Southern California Earthquake Center to characterize the earthquake hazards in a region. Sipht [28] which is used to automate the search for untranslated RNAs (sRNAs) for the bacterial replicons in the NCBI database, and LIGO [37] that is used to generate and analyze gravitational waveforms from data collected during the coalescing of compact binary systems. These scientific workflows are generated with 30 tasks by Bharathi et al [5,28].…”

An Integrated Scratch Management Service for HPC Centers

“…We have also begun developing a decentralized data delivery service for HPC applications. For our initial investigation, we have relied on both cloud [13] and non-cloud [12] resources, and PlanetLab [1] for investigating the effectiveness of our approach.…”

“…Approach feasibility using Azure: We have begun the integration of cloud and HPC by developing a data transfer service CATCH [13]. CATCH uses Azure [10] for cloud storage and FUSE [4] to provide HPC jobs a transparent file system mount point to access the cloud resources.…”

Towards Improving MapReduce Task Scheduling Using Online Simulation Based Predictions

“…In recent years, MapReduce/Hadoop [1] has emerged as the de facto model for big data applications, and is employed by industry [2], [3], [4], [5] and academia [6], [7], [8] alike. Improving the efficiency of Hadoop is therefore crucial.…”