CAP3: A Cloud Auto-Provisioning Framework for Parallel Processing Using On-Demand and Spot Instances

Abstract

Abstract-Cloud computing has drawn increasing attention from the scientific computing community due to its ease of use, elasticity, and relatively low cost. Because a high-performance computing (HPC) application is usually resource demanding, without careful planning, it can incur a high monetary expense even in Cloud. We design a tool called CAP 3 (Cloud AutoProvisioning framework for Parallel Processing) to help a user minimize the expense of running an HPC application in Cloud, while meeting the user-specified job deadline. Given an HPC application, CAP 3 automatically profiles the application, builds a model to predict its performance, and infers a proper cluster size that can finish the job within its deadline while minimizing the total cost. To further reduce the cost, CAP 3 intelligently chooses the Cloud's reliable on-demand instances or low-cost spot instances, depending on whether the remaining time is tight in meeting the application's deadline. Experiments on Amazon EC2 show that the execution strategy given by CAP 3 is costeffective, by choosing a proper cluster size and a proper instance type (on-demand or spot).

A Resilient Framework for Fault Handling in Web Service Oriented Systems

“…Meanwhile, more efficient selection algorithms will be investigated. In addition, we will adapt the framework to generate fault handling strategies for cloud services by combining our works [17], [18], [19], [20] in the fields of service computing and cloud computing.…”

Resource provisioning in Science Clouds: Requirements and challenges

“…There is a considerable amount of research works addressing cloud resource provisioning and scheduling from the user or consumer perspective . Some authors have studied how to implement hybrid provisioning of resources between several cloud providers, or even between different computing infrastructures such as grids and clouds .…”

Improving MapReduce Performance By Using A New Partitioner In YARN

“…(4) PIY optimizes the network traffic by decreasing the amount of the transmitted data located on nodes acting as both Mapper and Reducers. (5) We conduct a performance evaluation with PIY in YARN (Hadoop 2.6.0). Compared with some other popular strategies, PIY can reduce the execution time by 35.62% and 50.65% in homogeneous and heterogeneous Hadoop cluster, respectively.…”

“…In addition, many DataNodes act as both Mapper and Reducer [12]. If we can stay as many as intermediate <key,value> pairs on these DataNodes by the partition method in shuffle phase, it also furthest decrease the network traffic [5]. It's assumed that there are many <key,value> pairs corresponding to a special key on those DataNodes simultaneously.…”