A GridFTP Overlay Network Service

Rizk, P.; Kiddle, Cameron; Simmonds, Rob

doi:10.1109/icgrid.2006.310996

Cited by 8 publications

(2 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The GridFTP overlay network service [58] implements a specialized data storage interface (DSI) to achieve split-TCP functionality. The GridFTP client command is issued with source and destination URLs A/C and C/D to denote a transfer between end points A and D through nodes C and D. In [59], the authors have extended this effort to use previous transfers as a measure to use a particular node in the transfer overlay.…”

Section: Related Workmentioning

confidence: 99%

Timely Result-Data Offloading for Improved HPC Center Scratch Provisioning and Serviceability

Monti

Butt

Vazhkudai

2011

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Modern High-Performance Computing (HPC) centers are facing a data deluge from emerging scientific applications. Supporting large data entails a significant commitment of the high-throughput center storage system, scratch space. However, the scratch space is typically managed using simple "purge policies," without sophisticated end-user data services to balance resource consumption and user serviceability. End-user data services such as offloading are performed using point-to-point transfers that are unable to reconcile center's purge and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. Such inefficiencies can be prohibitive to sustaining high performance. In this paper, we address the above issues by designing a framework for the timely, decentralized offload of application result data. Our framework uses an overlay of userspecified intermediate and landmark sites to orchestrate a decentralized fault-tolerant delivery. We have implemented our techniques within a production job scheduler (PBS) and data transfer tool (BitTorrent). Our evaluation using both a real implementation and supercomputer job log-driven simulations show that: the offloading times can be significantly reduced (90.4 percent for a 5 GB data transfer); the exposure window can be minimized while also meeting center-user service level agreements.

show abstract

Section: Related Workmentioning

confidence: 99%

Timely Result-Data Offloading for Improved HPC Center Scratch Provisioning and Serviceability

Monti

Butt

Vazhkudai

2011

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…The GridFTP overlay network service [26], [27] implements a specialized data storage interface (DSI) to achieve split-TCP functionality. IBP [22] offers a data distribution infrastructure with a set of strategically placed resources (storage depots) to move data, and implement what is referred to as logistical networking.…”

Section: Related Workmentioning

confidence: 99%

CATCH: A Cloud-Based Adaptive Data Transfer Service for HPC

Monti

Butt

Vazhkudai

2011

2011 IEEE International Parallel &Amp; Distributed Processing Symposium

View full text Add to dashboard Cite

Abstract-Modern High Performance Computing (HPC)applications process very large amounts of data. A critical research challenge lies in transporting input data to the HPC center from a number of distributed sources, e.g., scientific experiments and web repositories, etc., and offloading the result data to geographically distributed, intermittently available endusers, often over under-provisioned connections. Such enduser data services are typically performed using point-to-point transfers that are designed for well-endowed sites and are unable to reconcile the center's resource usage and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. To overcome these inefficiencies, decentralized HPC data services are emerging as viable alternatives. In this paper, we develop and enhance such distributed data services by designing CATCH, a Cloud-based Adaptive data Transfer serviCe for HPC. CATCH leverages a bevy of cloud storage resources to orchestrate a decentralized data transport with fail-over capabilities. Our results demonstrate that CATCH is a feasible approach, and can help improve the data transfer times at the HPC center by as much as 81.1% for typical HPC workloads.

show abstract