Workflow Scheduling to Minimize Data Movement Using Multi-constraint Graph Partitioning

Abstract

Among scheduling algorithms of scientific workflows, the graph partitioning is a technique to minimize data transfer between nodes or clusters. However, when the graph partitioning is simply applied to a complex workflow DAG, tasks in each parallel phase are not always evenly assigned to computation nodes since the graph partitioning algorithm is not aware of edge directions that represent task dependencies. Thus, we propose a new method of task assignment based on Multi-Constraint Graph Partitioning. This method relates the dimension of weight vectors to the rank of a task phase defined by traversing the task graph. Our algorithm is implemented in the Pwrake workflow system and evaluated the performance of the Montage workflow using a computer cluster. The result shows that the file size accessed from remote nodes is reduced from 88% to 14% of the total file size accessed during the workflow and that the elapsed time is reduced by 31%.

Multiobjective Level-Wise Scientific Workflow Optimization in IaaS Public Cloud Environment

“…Single-objective optimization approaches are focused on minimizing the workflow makespan through scheduling [20,22,23] and task clustering approach [1]. However, single-objective optimization is not sufficient for data intensive workflows.…”

“…For example, Tanaka and Tatebe [20] proposed a data-aware scheduling strategy that reduces makespan by minimizing data movement between cluster nodes was presented. However, their strategy can only operate with homogeneous resources during workflow execution and is not suitable for use in cloud environments.…”

“…Epigenomics is a CPU-bound bioinformatics workflow with eight levels of workflow tasks. Montage is a collection of programs that combines multiple shots of astronomical images to generate a custom mosaic image [20]. CyberShake is a seismology workflow application that is used by the Southern California Earthquake Center to characterize earthquake hazards in a region [3].…”

Cost-aware challenges for workflow scheduling approaches in cloud computing environments: Taxonomy and opportunities

“…Consequently, this endeavor introduces a number of problems: (i) different users compete for resources within the cloud computing environment [15][16][17]; (ii) data needs to be transferred from one resource to another [18][19][20][21][22][23]; (iii) the inter-dependency between tasks introduces high communication cost and/or time [24][25][26][27][28][29][30]; and (iv) the trade-off between QoS of user requirements and the cost of workflow execution [1,10,13,[31][32][33][34]. However, this review targets execution cost-related problems which can possibly affect both service consumers and utility providers.…”

Extreme Big Data (EBD): Next Generation Big Data Infrastructure Technologies Towards Yottabyte/Year

“…The Pwrake workflow system executes a workflow written in Rakefile in parallel assigning processes on a compute node where the input data is stored. The Pwrake, moreover, allocates processes to minimize data transfer size among compute nodes during a whole workflow execution by multi-constraint graph partitioning [34], which can reduce data transfer size of intermediate data generated during the workflow execution.…”