ExPERT: Pareto-Efficient Task Replication on Grids and a Cloud

Abstract

Abstract-Many scientists perform extensive computations by executing large bags of similar tasks (BoTs) in mixtures of computational environments, such as grids and clouds. Although the reliability and cost may vary considerably across these environments, no tool exists to assist scientists in the selection of environments that can both fulfill deadlines and fit budgets. To address this situation, we introduce the ExPERT BoT scheduling framework. Our framework systematically selects from a large search space the Pareto-efficient scheduling strategies, that is, the strategies that deliver the best results for both makespan and cost. ExPERT chooses from them the best strategy according to a general, user-specified utility function. Through simulations and experiments in real production environments, we demonstrate that ExPERT can substantially reduce both makespan and cost in comparison to common scheduling strategies. For bioinformatics BoTs executed in a real mixed grid+cloud environment, we show how the scheduling strategy selected by ExPERT reduces both makespan and cost by 30%-70%, in comparison to commonlyused scheduling strategies.

A Periodic Portfolio Scheduler for Scientific Computing in the Data Center

“…Closest to our work, our own [8,16,17] and related [44][45][46]5] studies of multiple scheduling policies have emphasized the inability of any single policy to perform well under a wide yet realistic variety of scientific workloads.…”

“…The main trade-off in the creation of this set is between capability to schedule different workload patterns and application types, and time required to explore the set during the selection and application phases. The selection of policies is usually done by an expert, as we do in Section 3.3, but can also be done automatically, for example as the result of an automated comparative study of policies specific to one domain [8,16,17].…”

“…Although a lower slowdown is to be desired, it may be the result of (much) higher cost, for example when the provisioning policy is STU (StartUp in Section 3.3). To balance these considerations, we use an extension of an utility function, which is defined elsewhere [16,17]:…”

“…For the latter, recent work has focused on new layouts of networks [12,13] and new virtualization architectures [14,15]. Our previous scheduling studies [8,16,17], which evaluate a large variety of scheduling policies for various types of scientific computing, indicate that no single policy can accommodate all workload conditions, and all user and system objective functions. Thus, a tension arises in trying to select the appropriate resource scheduling policies for the data center.…”

Scheduling Jobs in the Cloud Using On-Demand and Reserved Instances

“…Academia [1][2][3][4][5][6][7] and industry [8] are both increasingly using cloud resources as infrastructure to serve their users, due to the elastic, flexible, and pay-as-you-go features of Infrastructureas-a-Service (IaaS) clouds. Cloud brokers need to lease resources from IaaS clouds cheaply, yet execute the users' jobs in time.…”