Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds

Malawski, Maciej; Juve, Gideon; Deelman, Ewa; Nabrzyski, Jarek

doi:10.1016/j.future.2015.01.004

Cited by 204 publications

(146 citation statements)

References 45 publications

Supporting

Mentioning

144

Contrasting

Unclassified

Order By: Relevance

“…The disparate processing units share no system resources, they have their own operating system, and communicate through high-speed network. The main computing models within the distributed parallel computing systems include cluster [26,89], grid [13,32,82,86], and cloud computing [33,59,82].…”

Section: Parallel Computing Systemsmentioning

confidence: 99%

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

et al. 2018

View full text Add to dashboard Cite

While modern parallel computing systems offer high performance, utilizing these powerful computing resources to the highest possible extent demands advanced knowledge of various hardware architectures and parallel programming models. Furthermore, optimized software execution on parallel computing systems demands consideration of many parameters at compile-time and run-time. Determining the optimal set of parameters in a given execution context is a complex task, and therefore to address this issue researchers have proposed different approaches that use heuristic search or machine learning. In this paper, we undertake a systematic literature review to aggregate, analyze and classify the existing software optimization methods for parallel computing systems. We review approaches that use machine learning or meta-heuristics for software optimization at compile-time and run-time. we discuss challenges and future research directions. The results of this study may help to better understand the state-of-the-art techniques that use machine learning and meta-heuristics to deal with the complexity of software optimization for parallel computing systems. Furthermore, it may aid in understanding the limitations of existing approaches and identification of areas for improvement.

show abstract

Section: Parallel Computing Systemsmentioning

confidence: 99%

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Malawski et al [8] presented a dynamic resource provisioning and scheduling algorithm called DPDS. It schedules workflows under given deadline and budget constraints along with the information about resource utilization for VM provisioning and scheduling.…”

Section: Related Workmentioning

confidence: 99%

“…For example, dynamic provisioning of resources is not considered in [5][6][7][8], scalability in terms of large number of tasks is not considered in [9], heterogeneity of resources is not considered in [8,10], resource auto-scaling is not considered in [11], data dependencies are not considered in [12] and task clustering technique in [5] is not fully autonomous. Moreover, unlike multiple independent BoTs or single task-based workflows, the concept of using multiple connected and constrained BoTs for reducing the data transfer time is not considered in most existing scheduling algorithms [13,14].…”

Section: Introductionmentioning

confidence: 99%

Elastic Scheduling of Scientific Workflows under Deadline Constraints in Cloud Computing Environments

Anwar

Deng

2018

Future Internet

View full text Add to dashboard Cite

Scientific workflow applications are collections of several structured activities and finegrained computational tasks. Scientific workflow scheduling in cloud computing is a challenging research topic due to its distinctive features. In cloud environments, it has become critical to perform efficient task scheduling resulting in reduced scheduling overhead, minimized cost and maximized resource utilization while still meeting the user-specified overall deadline. This paper proposes a strategy, Dynamic Scheduling of Bag of Tasks based workflows (DSB), for scheduling scientific workflows with the aim to minimize financial cost of leasing Virtual Machines (VMs) under a userdefined deadline constraint. The proposed model groups the workflow into Bag of Tasks (BoTs) based on data dependency and priority constraints and thereafter optimizes the allocation and scheduling of BoTs on elastic, heterogeneous and dynamically provisioned cloud resources called VMs in order to attain the proposed method's objectives. The proposed approach considers pay-asyou-go Infrastructure as a Service (IaaS) clouds having inherent features such as elasticity, abundance, heterogeneity and VM provisioning delays. A trace-based simulation using benchmark scientific workflows representing real world applications, demonstrates a significant reduction in workflow computation cost while the workflow deadline is met. The results validate that the proposed model produces better success rates to meet deadlines and cost efficiencies in comparison to adapted state-of-the-art algorithms for similar problems.

show abstract

“…In many scientific domains, large computational problems are frequently modeled using multiple inter-related workflows grouped into ensembles [79]. Typically, the workflows in an ensemble have similar structure, but may differ in their input data and parameters, such as the input data or the number of workflow tasks.…”

Section: Workflow Ensemblesmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

Self Cite

119

View full text Add to dashboard Cite

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

Algorithms for cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds

Cited by 204 publications

References 45 publications

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

Elastic Scheduling of Scientific Workflows under Deadline Constraints in Cloud Computing Environments

A characterization of workflow management systems for extreme-scale applications

Contact Info

Product

Resources

About