A Reliability Aware Algorithm for Workflow Scheduling on Cloud Spot Instances Using Artificial Neural Network

Ghavamipoor, Hoda; Mousavi, Sayyed Ali Kianian; Faragardi, Hamid Reza; Rasouli, Nayereh

doi:10.1109/ist50524.2020.9345896

Cited by 5 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that HPC@Cloud does not possess a predictive mechanism for the timing and frequency of spot evictions, we limit ourselves to estimating a lower‐bound for execution costs. This constraint exists as strategies for predicting spot failures, such as those presented by Ghavamipoor et al, 29 have yet to be integrated into HPC@Cloud .…”

Section: Resultsmentioning

confidence: 99%

“…Cost prediction. We intend to integrate existing solutions to estimate when a spot revocation may happen based on historical data into HPC@Cloud , such as those proposed by Ghavamipoor et al 29 Based on these predictions, the Resource Manager Module could trigger checkpoints when the probability of a spot instance revocations becomes high, thus being a generic alternative for provider‐specific alarm systems. These predictions could also be useful to cleverly choose instance types, or even know beforehand if using spot instances will be cost‐effective or not, basing the decision‐making on the discount rates, probabilities of eviction and estimated number of failures at runtime.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Munhoz,

Castro

2023

Concurrency and Computation

View full text Add to dashboard Cite

The advent of cloud computing has made access to computing infrastructure available to millions of users that face resource constraints. In the context of high performance computing (HPC), public cloud resources have emerged as a cost‐effective alternative to expensive on‐premises clusters. However, there are several challenges and limitations in adopting this approach. This paper proposes HPC@Cloud , a provider‐agnostic open‐source software toolkit that facilitates the migration, testing, and execution of HPC applications in public clouds. The toolkit takes advantage of various fault tolerance technologies to enable the use of inexpensive transient cloud infrastructure, commonly known as “spot” instances. Also, it features integration with singularity containers, allowing users to run complex applications on virtual HPC clusters in a portable and reproducible way. Finally, it provides a data‐based empirical approach to estimating cloud infrastructure costs for HPC workloads. The results obtained on two public cloud providers (AWS and Vultr) show that: (i) HPC@Cloud can efficiently build virtual HPC clusters on the cloud; (ii) the new adaptive fault tolerance strategy outperforms other existing strategies based on blocking restoration; (iii) the integration of singularity containers into HPC@Cloud improves the portability of HPC applications to public clouds with negligible performance penalty to the applications; (iv) the proposed cost prediction approach can estimate the cost of running the applications on AWS and Vultr with up to 93% accuracy on average.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Munhoz,

Castro

2023

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…The optimal solution is produced offline before execution, and actions selected on-the-fly depend on the occurrence of instance revocations and the actual task completion time. Ghavamipoor et al [31] employed an artificial neural network to define a failure prediction module for SIs, and proposed a reliability-aware scheduling algorithm for minimizing the makespan of workflows considering a minimum ensured reliability rate. Some studies combined multiple strategies to address the volatility when executing workflows in SIs.…”

Section: Workflow Scheduling In Volatile Cloudsmentioning

confidence: 99%

Monte Carlo Simulation-Based Robust Workflow Scheduling for Spot Instances in Cloud Environments

Wu,

Fang,

Zeng

et al. 2024

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

When deploying workflows in cloud environments, the use of Spot Instances (SIs) is intriguing as they are much cheaper than on-demand ones. However, SIs are volatile and may be revoked at any time, which results in a more challenging scheduling problem involving execution interruption and hence hinders the successful handling of conventional cloud workflow scheduling techniques. Although some scheduling methods for SIs have been proposed, most of them are no more applicable to the latest SIs, as they have evolved by eliminating bidding and simplifying the pricing model. This study focuses on how to minimize the execution cost with a deadline constraint when deploying a workflow on volatile SIs in cloud environments. Based on Monte Carlo simulation and list scheduling, a stochastic scheduling method called MCLS is devised to optimize a utility function introduced for this problem. With the Monte Carlo simulation framework, MCLS employs sampled task execution time to build solutions via deadline distribution and list scheduling, and then returns the most robust solution from all the candidates with a specific evaluation mechanism and selection criteria. Experimental results show that the performance of MCLS is more competitive compared with traditional algorithms.

show abstract

“…Conversely, for speci c cloud services, the current proposal outscored the heuristic approach by roughly 27%. When data was dispersed over four cloud storage providers, EAIFDE reduced latency by roughly 79 percent in comparison to Security Disk Detection and by about 67 percent as compared to the heuristic technique[20]. Compared to the OFDAMCSS technique, this solution saves roughly 61 percent.…”

mentioning

confidence: 97%

AI - Based Framework for Private Cloud Computing

Dhaya

Kanthavel

2021

Preprint

View full text Add to dashboard Cite

Artificial Intelligence (AI) systems are computational simulations that are "trained" using information and expert input to duplicate a professional's choice given the same data. Only using one private cloud storage service to store information can cause a variety of issues for the system administrator. Knowledge providers, scalability, efficiency, privacy, and the potential of vendor support are examples of such concerns. Distributing information across several cloud storage services, comparable to how data is dispersed between various physical disk drives to increase error detection and increase productivity, is a possible approach. Moreover, because multiple private cloud providers have varying pricing strategies and service quality, maximizing the efficiency and profitability of many cloud providers at the same time is difficult. Based on access permission behaviors, this study presents a methodology for dynamically modifying network management rules across several cloud providers. The goal of this research is to look into how to reduce both the estimated cost and delay periods for numerous cloud providers. The architecture was put into practice in a cloud storage systems emulator, which simulated the complexity and effectiveness of numerous cloud providers in a real-world context. In particular, the architecture was evaluated in a variety of cloud storage environments. The outcomes of the platform's testing revealed that many cloud methods were successful.

show abstract

A Reliability Aware Algorithm for Workflow Scheduling on Cloud Spot Instances Using Artificial Neural Network

Cited by 5 publications

References 18 publications

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Enabling the execution of HPC applications on public clouds with HPC@Cloud toolkit

Monte Carlo Simulation-Based Robust Workflow Scheduling for Spot Instances in Cloud Environments

AI - Based Framework for Private Cloud Computing

Contact Info

Product

Resources

About