A Periodic Portfolio Scheduler for Scientific Computing in the Data Center

Abstract

Abstract. The popularity of data centers in scientific computing has led to new architectures, new workload structures, and growing customerbases. As a consequence, the selection of efficient scheduling algorithms for the data center is an increasingly costlier and more difficult challenge. To address this challenge, and contrasting previous work on scheduling for scientific workloads, we focus in this work on portfolio schedulinghere, the dynamic selection and use of a scheduling policy, depending on the current system and workload conditions, from a portfolio of multiple policies. We design a periodic portfolio scheduler for the workload of the entire data center, and equip it with a portfolio of resource provisioning and allocation policies. Through simulation based on real and synthetic workload traces, we show evidence that portfolio scheduling can automatically select the scheduling policy to match both user and data center objectives, and that portfolio scheduling can perform well in the data center, relative to its constituent policies.

Exploring portfolio scheduling for long-term execution of scientific workloads in IaaS clouds

“…To balance these considerations, we use an extension of a utility function defined in prior work [2,8,41]:…”

“…For this metric, κ is a scaling factor for the total utility, which we set to 100 as in our prior work [8]. The metric parameters α and β are used to express different utility functions: α is used to emphasize the efficiency of resource usage and β is used to stress the urgency of the jobs.…”

“…The goal of this step is to find a mapping function S such that the selected scheduling policy a ∈ A is able to optimize the given performance metric y ∈ Y for the current workload x ∈ P . Our previous work [8] focused on this problem and solved it by using online simulation as the selection mapping function S(·). Nonetheless, online simulation is a time-consuming function, especially when there are hundreds of jobs in the workload and tens of policies in the portfolio.…”

“…Similarly to our previous work [8,41], we break the scheduling policies into three parts: the provisioning policies, the job selecting policies and the VM selecting policies. To run the queued jobs, the scheduling policy first provisions a suitable number of VMs from the cloud through the resource provisioning policy, then uses the job selecting policy to order the queued jobs; for each selected job, it selects the required number of appropriate VMs from the cloud according to the VM selecting policy.…”

“…Instead of striving to find better individual scheduling policies, our previous research [8] adapted portfolio scheduling [12] to scientific workloads running in IaaS clouds. The basic idea of portfolio scheduling is to construct a portfolio of scheduling policies, and to select from it the most suitable policy for the current workload and system conditions.…”