SpotMPI: A Framework for Auction-Based HPC Computing Using Amazon Spot Instances

Taifi, Moussa; Shi, Justin Y.; Khreishah, Abdallah

doi:10.1007/978-3-642-24669-2_11

Cited by 29 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Their later research employs fault tolerance such as checkpointing, task duplication, and migration [25]. SpotMPI [26] facilitates execution of MPI applications on auction-based Cloud platforms based on adjusted optimal checkpoint-restart (CPR) intervals. Their tookit can automate checkpointing at bidding price and restart application after interruption.…”

Section: Related Workmentioning

confidence: 99%

CAP3: A Cloud Auto-Provisioning Framework for Parallel Processing Using On-Demand and Spot Instances

Huang

Wang

Tak

et al. 2013

2013 IEEE Sixth International Conference on Cloud Computing

View full text Add to dashboard Cite

Abstract-Cloud computing has drawn increasing attention from the scientific computing community due to its ease of use, elasticity, and relatively low cost. Because a high-performance computing (HPC) application is usually resource demanding, without careful planning, it can incur a high monetary expense even in Cloud. We design a tool called CAP 3 (Cloud AutoProvisioning framework for Parallel Processing) to help a user minimize the expense of running an HPC application in Cloud, while meeting the user-specified job deadline. Given an HPC application, CAP 3 automatically profiles the application, builds a model to predict its performance, and infers a proper cluster size that can finish the job within its deadline while minimizing the total cost. To further reduce the cost, CAP 3 intelligently chooses the Cloud's reliable on-demand instances or low-cost spot instances, depending on whether the remaining time is tight in meeting the application's deadline. Experiments on Amazon EC2 show that the execution strategy given by CAP 3 is costeffective, by choosing a proper cluster size and a proper instance type (on-demand or spot).

show abstract

Section: Related Workmentioning

confidence: 99%

CAP3: A Cloud Auto-Provisioning Framework for Parallel Processing Using On-Demand and Spot Instances

Huang

Wang

Tak

et al. 2013

2013 IEEE Sixth International Conference on Cloud Computing

View full text Add to dashboard Cite

show abstract

“…While a set of published projects are setting the tone for future developments in terms of resource planning and management [37], since the problem space is so large, we still expect many new approaches to meet the HPC sustainability challenges: performance, reliability, scalability, intercloud operability, and usability of virtual clusters all at the same time.…”

Section: Challenges Of Virtual Cluster Managementmentioning

confidence: 99%

Building a Private HPC Cloud for Compute and Data-Intensive Applications

Taifi¹,

Khreishah²,

Shi³

2013

IJCCSA

Self Cite

View full text Add to dashboard Cite

show abstract

“…While these research efforts have been successful at helping clarify many points about spot instances, only a few, such as [22], have touched on HPC applications and how the difference between high performance and high throughput applications impacts the fault tolerance strategies used to mitigate the interruptions due to market fluctuations. While describing strategies for single applications is crucial to the understanding of spot resources and auction-based computing in general, this knowledge is not fully transferable to HPC computing and there are many differences when applications are meant to scale to higher orders of magnitudes.…”

Section: Previous Work On Spot Instancesmentioning

confidence: 99%

Banking on decoupling

Taifi

2013

SIGOPS Oper. Syst. Rev.

Self Cite

View full text Add to dashboard Cite

Cloud providers are auctioning their excess capacity using dynamically priced virtual instances. These spot instances provide significant savings compared to on-demand or fixed price instances. The users willing to use these resources are asked to provide a maximum bid price per hour, and the cloud provider runs the instances as long as the market price is below the user's bid price. By using such resources, the users are exposed explicitly to failures, and need to adapt their applications to provide some level of fault tolerance. In this paper, we expose the effect of bidding in the case of virtual HPC clusters composed of spot instances. We describe the interesting effect of uniform versus non-uniform bidding in terms of both the failure rate and the failure model. We propose an initial attempt to deal with the problem of predicting the runtime of a parallel application under various bidding strategies and various system parameters. We describe the relationship between bidding strategies and programming models, and we build a preliminary optimization model that uses real price traces from Amazon Web Services as inputs, as well as instrumented values related to the processing and network capacities of cluster instances on the EC2 services. Our results show preliminary insights into the relationship between non-uniform bidding and application scaling strategies.

show abstract

SpotMPI: A Framework for Auction-Based HPC Computing Using Amazon Spot Instances

Cited by 29 publications

References 16 publications

CAP3: A Cloud Auto-Provisioning Framework for Parallel Processing Using On-Demand and Spot Instances

CAP3: A Cloud Auto-Provisioning Framework for Parallel Processing Using On-Demand and Spot Instances

Building a Private HPC Cloud for Compute and Data-Intensive Applications

Banking on decoupling

Contact Info

Product

Resources

About