Scheduling bulk file transfers with start and end times

Abstract

The advancement of optical networking technologies has enabled e-science applications that often require transport of large volumes of scientific data. In support of such data-intensive applications, we develop and evaluate control plane algorithms for scheduling bulk file transfers, where each transfer has a start time and an end time. We formulate the scheduling problem as a special type of the multi-commodity flow problem. To cope with the start and end time constraints of the file-transfer jobs, we divide time into uniform time slices. Bandwidth is allocated to each job on every time slice and is allowed to vary from slice to slice. This enables periodical adjustment of the bandwidth assignment to the jobs so as to improve a chosen performance objective: throughput of the concurrent transfers. In this paper, we study the effectiveness of using multiple time slices, the performance criterion being the tradeoff between achievable throughput and the required computation time. Furthermore, we investigate using multiple paths for each file transfer to improve the throughput. We show that using a small number of paths per job is generally sufficient to achieve near optimal throughput with a practical execution time, and this is significantly higher than the throughput of a simple scheme that uses single shortest path for each job.

Algorithms and Stability Analysis for Content Distribution over Multiple Multicast Trees

“…If the time budget for tree computation is chosen to be very short, e.g., close to the round-trip propagation time, we may restrict each multicast session to use a small number of precomputed trees (e.g., 4 -100 trees); finding the min-cost tree among them is trivial. Experiences have shown that the loss in throughput due to the restriction on the trees is likely to be small (see [38] for the unicast case). The performance guarantees found in the paper still apply if we modify the optimization problem by adding the constraints about the allowed trees.…”

Advance Resource Provisioning in Bulk Data Scheduling

“…Data transfer scheduling with a specific start time and a particular deadline has been studied in [34,38,35]. The scheduling problem has been formulated as a multicommodity flow problem, and uniform time slices have been used to model the time dependency in [35].…”

“…The scheduling problem has been formulated as a multicommodity flow problem, and uniform time slices have been used to model the time dependency in [35]. The objective is to maximize the total transfer throughput, and data transfers can use varying bandwidth in every time slice.…”

An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers

“…Data transfer scheduling with a specific start time and a particular deadline has been studied in [39,47,42]. The scheduling problem has been formulated as a multi-commodity flow problem, and uniform time slices have been used to model the time dependency in [42].…”

“…The scheduling problem has been formulated as a multi-commodity flow problem, and uniform time slices have been used to model the time dependency in [42]. The objective is to maximize the total transfer throughput and data transfers can use varying bandwidth in every time slice.…”