The practical obstacles of data transfer

Abstract

The importance of computing facilities is heralded every six months with the announcement of the new Top500 list, showcasing the world's fastest supercomputers. Unfortunately, with great computing capability does not come great long-term data storage capacity, which often means users must move their data to their local site archive, to remote sites where they may be doing future computation or analysis, or back to their home institution, else face the dreaded data purge that most HPC centers employ to keep utilization of large parallel filesystems low to manage performance and capacity. At HPC centers, data transfer is crucial to the scientific workflow and will increase in importance as computing systems grow in size. The Energy Sciences Network (ESnet) recently launched its fifth generation network, a 100 Gbps high-performance, unclassified national network connecting more than 40 DOE research sites to support scientific research and collaboration. Despite the tenfold increase in bandwidth to DOE research sites amenable to multiple data transfer streams and high throughput, in practice, researchers often under-utilize the network and resort to painfully-slow single stream transfer methods such as scp to avoid the complexity of using multiple stream tools such as GridFTP and bbcp, and contend with frustration from the lack of consistency of available tools between sites. In this study we survey and assess the data transfer methods provided at several DOE supported computing facilities, including both leadership-computing facilities, connected through ESnet. We present observed transfer rates, suggested optimizations, and discuss the obstacles the tools must overcome to receive wide-spread adoption over scp.

Transfer of Large Volume Data over Internet with Parallel Data Links and SDN

“…Now more and more scientific and business sectors are dealing (or plan to) with the "Big data" [2-7, 33]. The interest to data transfer of increasing volumes is growing [11,12].…”

“…Such the testbed developm work laboratory [28]. The need for testbed is becoming surement results performed by authors [11]. The authors ents for one data transfer stream and many data streams.…”

“…It was discovered that no change in the tran ms more than 8. At the same time in [11] there is no in nel parameters and other details of measurement. is planned to get precise answers on those questions.…”