Instant GridFTP

Abstract

A foundational need in high-performance computing is to move large (multi-gigabyte and even terabyte) datasets between sites. Simple file transfer mechanisms such as FTP and SCP are not sufficient from either a reliability or performance perspective. GridFTP is the de facto standard protocol for transferring large data files in production Grid/HPC environments. GridFTP extends the standard FTP protocol to provide a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The Globus GridFTP implementation has become the preeminent highperformance data transfer tool for the Grid community, with large facilities and projects using it to transfer billions of files per year. We report here on a new product, Globus Connect Multi User (GCMU), that greatly streamlines Globus GridFTP installation and configuration. GCMU packages a GridFTP server, MyProxy Online Certificate Authority, and other components in a manner that avoids the need for any end-user or system administrator involvement in security configuration or credential management. We describe the GCMU design and a GridFTP protocol extension that simplifies transfers across security domains. We also explain how GCMU interacts with the Globus Online software-as-a-service solution. By enabling "instant GridFTP," this work makes the powerful Globus GridFTP tool accessible to nonexpert users and to smaller laboratories and projects.

Model-driven multisite workflow scheduling

“…Swift provides a simple reactive resource scheduling wherein, based on an initial "wave" of jobs, it records the per site job completion rate and adjusts the propor tionate number of jobs to be sent to these sites. Even though Swift can use GridFTP [4], [5] for high-speed data movement, it does not take data transfer time into account while picking the sites for executing the tasks. Our approach takes data transfer time into account while picking the execution sites.…”