Scheduling tasks sharing files on heterogeneous master–slave platforms

Abstract

This paper is devoted to scheduling a large collection of independent tasks onto heterogeneous clusters. The tasks depend upon (input) files which initially reside on a master processor. A given file may well be shared by several tasks. The role of the master is to distribute the files to the processors, so that they can execute the tasks. The objective for the master is to select which file to send to which slave, and in which order, so as to minimize the total execution time. The contribution of this paper is twofold. On the theoretical side, we establish complexity results that assess the difficulty of the problem. On the practical side, we design several new heuristics, which are shown to perform as efficiently as the best heuristics in [H. Casanova, A. Legrand, D. Zagorodnov, F. Berman, Heuristics for scheduling parameter sweep applications in Grid environments, in: Ninth Heterogeneous Computing Workshop, IEEE Computer Society Press, Silver Spring, MD, 2000, pp. 349-363; Berman, Using simulation to evaluate scheduling heuristics for a class of applications in Grid environments, Research Report RR-1999-46, LIP, ENS Lyon, France, 1999 although their cost is an order of magnitude lower.

Improving scalability of Bag-of-Tasks applications running on master–slave platforms

“…BoT applications composed of independent tasks with file sharing have been studied in several papers during the last years [4,6,[8][9][10][11]15,[17][18][19]. Their relevance has motivated the development of specialized environments which aim to facilitate the execution of large BoT applications on computational grids and clusters, such as the AppLeS Parameter-Sweep Template (APST) [3] and MyGrid [7].…”

“…This improvement allowed for a reduction of the bottleneck in the master processor. In [8], Giersch et al proved theoretical limits for the computational complexity associated to the scheduling problem. The authors also proposed several new heuristics which produce schedules that approach the quality achieved by the heuristics proposed by Casanova et al [4], while keeping the computational complexity one order of magnitude faster.…”

“…The authors also proposed several new heuristics which produce schedules that approach the quality achieved by the heuristics proposed by Casanova et al [4], while keeping the computational complexity one order of magnitude faster. In [11], Kaya and Aykanat proposed an iterative scheduling approach that produces effective and efficient schedules, compared to the previous works in [4,8]. The non-oblivious algorithm proposed by Kaya and Aykanat has shown excellent performance for heterogeneous platforms, outperforming well known non-oblivious algorithms such as Sufferage.…”

“…All tasks and files have the same size, and each task depends upon a single non-shared file. Note that the problem of scheduling an BoT application where each task depends upon a single nonshared file, all tasks and files have the same size and the master-slave platform is homogeneous, has polynomial complexity [8], with the optimal scheduling algorithm being the simple greedy round-robin algorithm. These assumptions are considered in the analysis that follows.…”

“…Each task depends on one or more input files for execution, and input files can be shared among tasks. Generally a BoT application can be represented as a bipartite graph where one set of nodes are input files, the other set are composed of tasks and the links represent dependencies among tasks and files [8] (see Fig. 1).…”