Dynamic Adaptation of Parallelism Level in Data Transfer Scheduling

Williams

et al. 2010

Self Cite

Many scientific applications and experiments, such as high energy and nuclear physics, astrophysics, climate observation and modeling, combustion, nano-scale material sciences, and computational biology, generate extreme volumes of data with a large number of files. These data sources are distributed among national and international data repositories, and are shared by large numbers of geographically distributed scientists. A large portion of data is frequently accessed, and a large volume of data is moved from one place to another for analysis and storage. One challenging issue in such efforts is the limited network capacity for moving large datasets to explore and manage. The Bulk Data Mover (BDM), a data transfer management tool in the Earth System Grid (ESG) community, has been managing the massive dataset transfers efficiently with the pre-configured transfer properties in the environment where the network bandwidth is limited. Dynamic transfer adjustment was studied to enhance the BDM to handle significant end-to-end performance changes in the dynamic network environment as well as to control the data transfers for the desired transfer performance. We describe the results from the BDM transfer management for the climate datasets. We also describe the transfer estimation model and results from the dynamic transfer adjustment.

“…Dynamically setting the number of optimal parallel streams has been introduced in [27]. Further, there are several studies in adaptive parameter tuning [20,22].…”

Section: Application-level Dynamic Tuningmentioning

confidence: 99%

Bulk Data Movement for Climate Dataset: Efficient Data Transfer Management with Dynamic Transfer Adjustment

Sim

Williams

et al. 2010

Self Cite

“…A common approach to increase overall throughput is to use parallel streams, so that multiple threads (and CPU cores) work si-multaneously to overcome the latency cost generated by disk and memory copy operation in the end system. Another approach is to use concurrent transfers, where multiple transfer tasks cooperate together to generate high throughput data in order to fill the network pipe [25,4]. In standard file transfer mechanisms, we need more parallelism to overcome the cost of bookkeeping and control messages.…”

Section: Climate Data Movermentioning

confidence: 99%

Experiences with 100Gbps network applications

Proceedings of the Fifth International Workshop on Data-Intensive Distributed Computing Date

Pouyoul

Yao

et al. 2012

Self Cite

Abstract100Gbps networking has finally arrived, and many research and educational institutions have begun to deploy 100Gbps routers and services. ESnet and Internet2 worked together to make 100Gbps networks available to researchers at the Supercomputing 2011 conference in Seattle Washington. In this paper, we describe two of the first applications to take advantage of this network. We demonstrate a visualization application that enables remotely located scientists to gain insights from large datasets. We also demonstrate climate data movement and analysis over the 100Gbps network. We describe a number of application design issues and host tuning strategies necessary for enabling applications to scale to 100Gbps rates.

“…The achievable end-to-end throughput and the system load in communicating parties might change during the period of a data transfers, especially when large volume of data needs to be transmitted. Therefore, dynamic approaches in which data transfer tuning is performed on the fly [6], are highly desirable in order to adapt to varying environmental conditions to come up with a high-quality tuning for best system and network utilization.…”

Section: Open Challenges In Ndmmentioning

confidence: 99%

Open problems in network-aware data management in exa-scale computing and terabit networking era

Proceedings of the First International Workshop on Network-Aware Data Management

Byna

2011

Self Cite

Accessing and managing large amounts of data is a great challenge in collaborative computing environments where resources and users are geographically distributed. Recent advances in network technology led to next-generation highperformance networks, allowing high-bandwidth connectivity. Efficient use of the network infrastructure is necessary in order to address the increasing data and compute requirements of large-scale applications. We discuss several open problems, evaluate emerging trends, and articulate our perspectives in network-aware data management.