Evaluating Cloud Computing Techniques for Smart Power Grid Design Using Parallel Scripting

Future Generation Computer Systems

Jung

Meng

et al. 2016

Please cite this article as: K. Maheshwari, E.-S. Jung, J. Meng, V. Morozov, V. Vishwanath, R. Kettimuthu, Workflow performance improvement using model-based scheduling over multiple clusters and clouds, Future Generation Computer Systems (2015), http://dx. AbstractIn recent years, a variety of computational sites and resources have emerged, and users often have access to multiple resources that are distributed. These sites are heterogeneous in nature and performance of different tasks in a workflow varies from one site to another. Additionally, users typically have a limited resource allocation at each site capped by administrative policies. In such cases, judicious scheduling strategy is required in order to map tasks in the workflow to resources so that the workload is balanced among sites and the overhead is minimized in data transfer. Most existing systems either run the entire workflow in a single site or use naïve approaches to distribute the tasks across sites or leave it to the user to optimize the allocation of tasks to distributed resources. This results in a significant loss in productivity. We propose a multi-site workflow scheduling technique that uses performance models to predict the execution time on resources and dynamic probes to identify the achievable network throughput between sites. We evaluate our approach using real world applications using the Swift parallel and distributed execution framework. We use two distinct computational environments-geographically distributed multiple clusters and multiple clouds. We show that our approach improves the resource utilization and reduces execution time when compared to the default schedule.

Section: Swift: Parallel Workflow Scriptingmentioning

confidence: 99%

“…Swift is a workflow framework for parallel execution of ordinary programs [12]. The Swift runtime contains a powerful platform for running user programs on a broad range of computational infrastructures, such as clouds, grids, clusters, and supercomputers out of the box.…”

Section: Swift: Parallel Workflow Scriptingmentioning

confidence: 99%

Workflow performance improvement using model-based scheduling over multiple clusters and clouds

Future Generation Computer Systems

Jung

Meng

et al. 2016

“…The queue wait times have been replaced by the time it takes virtual machines (VMs) to become available. (Our past experiments show that this is generally under one minute on popular clouds such as EC2 and is constant regardless of the requested number of VMs [13]). …”

Section: Introductionmentioning

confidence: 98%

Evaluating storage systems for scientific data in the cloud

Proceedings of the 5th ACM Workshop on Scientific Cloud Computing

Wozniak

Yang

et al. 2014

Infrastructure-as-a-Service (IaaS) clouds are an appealing resource for scientific computing. However, the bare-bones presentation of raw Linux virtual machines leaves much to the application developer. For many cloud applications, effective data handling is critical to efficient application execution. This paper investigates the capabilities of a variety of POSIX-accessible distributed storage systems to manage data access patterns resulting from workflow application executions in the cloud. We leverage the expressivity of the Swift parallel scripting framework to benchmark the performance of a number of storage systems using synthetic workloads and three real-world applications. We characterize two representative commercial storage systems (Amazon S3 and HDFS, respectively) and two emerging research-based storage systems (Chirp/Parrot and MosaStore). We find the use of aggregated node-local resources effective and economical compared with remotely located S3 storage. Our experiments show that applications run at scale with MosaStore show up to 30% improvement in makespan time compared with those run with S3. We also find that storage-system driven application deployments in the cloud results in better runtime performance compared with an on-demand datastaging driven approach.

“…Swift [1] is an application level scripting framework designed for composing ordinary programs into parallel applications. Applications encoded in U.S. Government work not protected by U.S. copyright Swift have been shown to execute on multiple computational sites via Swift coasters [2], [3] mechanism that implements the pilot jobs paradigm. Swift provides a simple reactive resource scheduling wherein, based on an initial "wave" of jobs, it records the per site job completion rate and adjusts the propor tionate number of jobs to be sent to these sites.…”

Section: Introductionmentioning

confidence: 99%

Model-driven multisite workflow scheduling

2013 IEEE International Conference on Cluster Computing (CLUSTER)

Jung

Meng

et al. 2013

Abstract-Workflows continue to play an important role in expressing and deploying scientific applications. In recent years, a wide variety of computational sites have emerged with shared access to users. A user may not be able to complete a complex workflow at a single site. It is thus beneficial to run different tasks of a workflow on different sites. For such cases, judicious scheduling strategy is required in order to map tasks in the workflow to resources at multiple sites so that the workload is balanced among sites and the overhead is minimized in data transfer. The key challenge is that the data transfer rate among sites varies based on the network capacity and load.We propose a workflow scheduling technique that tackles the multi-site task distribution challenge by using data movement performance modeling. We applied this technique to schedule an earth observation science workflow over three sites. Executed via the Swift parallel scripting paradigm, we augmented its default schedule and improved the time-to-completion by up to 52%.