Scalable Bulk Data Transfer in Wide Area Networks

Abstract

Bulk data transfer in wide area networks (WAN) requires scalable and high network bandwidth. In this paper, we identify a number of the scalability limitations that affect the full utilization of peak theoretical network bandwidth. In addition, we study and classify different offered approaches to overcome some of the identified limitations and increase network bandwidth among Grid components in WAN. With these limitations in mind, we study and evaluate the scalability and flexibility of a UDP-based multiple-network-interface socket (MuniSocket) model in WAN. The MuniSocket model is a middleware layer between the distributed applications and the multiple networks and system resources available. MuniSocket utilizes existing system resources, network interface cards, and network links to provide a scalable, reliable and high bandwidth network solution for data-intensive distributed applications, thus eliminating most of the identified limitations.

Self-configured multiple-network-interface socket

“…MuniSocket has been proposed to be used in a number of applications that require huge data transfer. Examples of these applications are a high throughput and scalable file system (Wei et al, 2005) and high-throughput communication for Grid computing (Burger et al, 2005;Mohamed et al, 2003). The original design of MuniSocket requires user intervention to setup and utilize the available channels and cannot automatically adapt to changes in the environment or the data flow.…”

High-performance message striping over reliable transport protocols

“…Example of unreliable transport service is that provided by UDP socket. In [11] and [14], we proposed a UDP-based protocol for MuniSocket to utilize the existing multiplenetwork interfaces and single or multiple networks to transfer large messages in both local and wide area networks. In order to have a reliable transfer service with the UDP-based protocol, the protocol itself included a reliability mechanism over the unreliable multiple services.…”

Dual direction big data download and analysis

“…Based on this set up we can have multiple streams downloading the data from different locations which reduces the download time significantly and overcomes some of the problem that may be encountered on the networks in use. In addition, as DDFTP uses TCP as its base protocol, opening multiple streams within the same data store, will also compensate for the limitations imposed due to the TCP error and flow control mechanisms [12] thus increasing the utilization of the network interfaces. As for the compute nodes, it will be more efficient as the data received is from opposite directions, thus each processing unit will be assigned a different processing direction on the data and parallelization can be enhances as there will be less contention.…”