Scheduling Bulk File Transfers with Start and End Times

Abstract

The advancement of optical networking technologies has enabled e-science applications that often require transport of large volumes of scientific data. In support of such data-intensive applications, we develop and evaluate control plane algorithms for scheduling bulk file transfers, where each transfer has a start time and an end time. We formulate the scheduling problem as a special type of the multi-commodity flow problem. To cope with the start and end time constraints of the file-transfer jobs, we divide time into uniform time slices. Bandwidth is allocated to each job on every time slice and is allowed to vary from slice to slice. This enables periodical adjustment of the bandwidth assignment to the jobs so as to improve a chosen performance objective: throughput of the concurrent transfers. In this paper, we study the effectiveness of using multiple time slices, the performance criterion being the tradeoff between achievable throughput and the required computation time. Furthermore, we investigate using multiple paths for each file transfer to improve the throughput. We show that using a small number of paths per job is generally sufficient to achieve near optimal throughput with a practical execution time, and this is significantly higher than the throughput of a simple scheme that uses single shortest path for each job.

Bandwidth On-Demand for Multimedia Big Data Transfer Across Geo-Distributed Cloud Data Centers

“…Big data transfer across cloud data centers is investigated in [14] and [15]. In [2], the study focuses on coordinating bulk data transfer between interdata centers while taking into consideration the priority of the transfer.…”

“…However, NetStitcher does not solve the issue of peak requests resulting from multiple data transfers with varying deadlines and priorities. The studies in [13], [14] and [15] although are not concerned with data transfer across geo-distributed cloud data centers, they address a similar issue related to deadline guaranteed of multiple bulk data transfers in grid networks. For example, [14] formulates the problem as a maximum concurrent multiple flows model and try to improve the throughput of transfers.…”

“…The studies in [13], [14] and [15] although are not concerned with data transfer across geo-distributed cloud data centers, they address a similar issue related to deadline guaranteed of multiple bulk data transfers in grid networks. For example, [14] formulates the problem as a maximum concurrent multiple flows model and try to improve the throughput of transfers. The study in [15] extends [14] to include a flexible admission control and periodic scheduling framework to blocking ratio of transfer requests.…”

Efficient algorithms for scheduling multiple bulk data transfers in inter‐datacenter networks

“…Rajah et al . propose a multi‐slice scheduling framework for multiple bulk transfers. They formulate the problem as a maximum concurrent multiple flow model and try to improve the throughput of transfers.…”

“…In their following work [33], they propose a flexible multi-interval resource reservation approach to improve the utilization of network, so that bulk transfers can be completed within less time. Rajah et al [34] propose a multi-slice scheduling framework for multiple bulk transfers. They formulate the problem as a maximum concurrent multiple flow model and try to improve the throughput of transfers.…”