Workflow performance improvement using model-based scheduling over multiple clusters and clouds

Maheshwari, Ketan; Jung, Eun-Sung; Meng, Jiayuan; Morozov, Vitali; Vishwanath, Venkatram; Kettimuthu, Rajkumar

doi:10.1016/j.future.2015.03.017

Cited by 32 publications

(19 citation statements)

References 38 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Makespan is referred to be the time length from the time that workflow execution starts to the time when its execution is finished. Many algorithms are proposed for the makespan minimization . Deadline is a common constraint combined with cost minimization of the workflow execution, which is the maximum makespan accepted by users.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Decentralized executions of privacy awareness data analytics workflows in the cloud

Yao

Cao

Qian

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

Summary Nowadays, with the development of cloud computing technology, an increasing number of enterprises or organizations have migrated applications to the cloud environment. Because of privacy concerns, a company may store sensitive data on a local server or a private cloud. As a result, the data analytics tasks have to be performed only in the local environment even when the analysis results will be shared to the outside. In this paper, we design a decentralized workflow system to speed up the execution of a Privacy‐Awareness Data Analytics application (PADA). Specifically, a workflow system with actor layer and engine layer for distributed execution of PADA proposed. The actors are responsible for performing actual data analysis tasks on the data side while the engines are deployed into different regions of the public cloud and communicate with actors to execute the whole workflow. Since engines in different regions have different communication latencies when invoking an actor, the optimization engine deployment scheme should be found to minimize the total execution time of a PADA. A path search‐based heuristic algorithm is designed to select suitable cloud regions to place the engines in the public cloud. The experimental results indicate that the proposed algorithm is effective in reducing application makespan.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Many algorithms are proposed for the makespan minimization. 11,18 Deadline is a common constraint combined with cost minimization of the workflow execution, 19,20 which is the maximum makespan accepted by users.…”

Section: Related Workmentioning

confidence: 99%

Decentralized executions of privacy awareness data analytics workflows in the cloud

Yao

Cao

Qian

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…The same approach is adopted by [37] and [101]. Regarding the execution of large-scale applications on similar scale systems, [68] suggest a multi-site workflow scheduling technique to enhance the range of available resources to execute workflows. While their approach does consider data transfers and the costs of sending data over expensive (slower) links that connect different geographically distributed sites, their approach does not consider 1) performance fluctuations during execution of the workflow, which would suggest the implementation of rescheduling and rebalancing mechanisms; 2) reliability mechanisms to cope with performance fluctuations due to failures; and 3) the influence of contention in the general I/O operations, such as sequential accesses to the same data inputs.…”

Section: Hybrid and Multicloud Scenariosmentioning

confidence: 99%

Cloud resource management: towards efficient execution of large-scale scientific applications and workflows on complex infrastructures

2017

View full text Add to dashboard Cite

Cloud computing evolved from the concept of utility computing, which is defined as the provision of computational and storage resources as a metered service. Another key characteristic of cloud computing is multitenancy, which enables resource and cost sharing among a large pool of users. Characteristics such as multitenancy and elasticity perfectly fit the requirements of modern data-intensive research and scientific endeavors. In parallel, as science relies on the analysis of very large data sets, data management and processing must be performed in a scalable and automated way. Workflows have emerged as a way to formalize and structure data analysis, thus becoming an increasingly popular paradigm for scientists to handle complex scientific processes. One of the key enablers of this conjunction of cloud computing and scientific workflows is resource management. However, several issues related to data-intensive loads, complex infrastructures such as hybrid and multicloud environments to support large-scale execution of workflows, performance fluctuations, and reliability, pose as challenges to truly position clouds as viable high-performance infrastructures for scientific computing. This paper presents a survey on cloud resource management that provides an extensive study of the field. A taxonomy is proposed to analyze the selected works and the analysis ultimately leads to the definition of gaps and future challenges to be addressed by research and development.

show abstract

“…Their work mainly concentrated on energy alone and need to calculate monetary cost connected with the use of VMs and profits for service providers. Maheshwari et al [19] presented performance models used multi-site workflow scheduling technique to calculate execution time on resources and active probes to categorize the attainable network throughput connecting sites without considering optimization parameters. Arabnejad et al [20] presented a heuristic scheduling algorithm considering quadratic time complexity including the two constraints time and cost but they failed to give importance to the runtime environment.…”

Section: Introductionmentioning

confidence: 99%

MCAMC: minimizing the cost and makespan of cloud service using non-dominated sorting genetic algorithm-II

2019

View full text Add to dashboard Cite

The number of cloud users and their aspiration for completion of tasks at less energy consumption and operating cost are rapidly increasing. Hence, the authors of this paper aim to minimize the makespan and operating cost by optimally scheduling the tasks and allocating the resources of cloud service. The optimum task scheduling and resource allocation are obtained for each objective function using the simple genetic algorithm. Further, the non-dominated solutions of the dual objectives are obtained using the non-dominated sorting genetic algorithm-II, the most successful multi-objective optimization technique. A complex cloud service problem consisting of ten tasks, fifteen subtasks and fifteen heterogeneous resources is considered to investigate the proposed method. The numerical results obtained in the single objective and multi objective optimization problems show that the makespan and the operating cost are significantly reduced using the simple genetic algorithm and a wide range of non-dominated solutions are obtained in the multi-objective optimization problem, by which the cloud users shall be benefitted to choose the most appropriate solution based on the other design constraints they have.

show abstract

Workflow performance improvement using model-based scheduling over multiple clusters and clouds

Cited by 32 publications

References 38 publications

Decentralized executions of privacy awareness data analytics workflows in the cloud

Decentralized executions of privacy awareness data analytics workflows in the cloud

Cloud resource management: towards efficient execution of large-scale scientific applications and workflows on complex infrastructures

MCAMC: minimizing the cost and makespan of cloud service using non-dominated sorting genetic algorithm-II

Contact Info

Product

Resources

About