Towards an MPI-Like Framework for the Azure Cloud Platform

Abstract

Message Passing Interface (MPI) has been the predominant standardized system for writing parallel and distributed applications. However, while MPI has been the software system of choice for traditional parallel and distributed computing platforms such as large compute clusters and Grid, MPI is not the system of choice for cloud platforms. The primary reasons for this is the lack of low latency high bandwidth network capabilities of the cloud platforms and the inherent architectural differences from traditional compute clusters. Prior studies suggest that the message latency of cloud platforms could be as much as 35x slower than that of an infiniband-connected cluster [1] for popular MPI implementations. MPI-like environment on cloud platforms is desirable for a large class of applications that run for long time spans with varying computing needs, such as the modeling and analysis to predict swath of a hurricane. Such applications could benefit from cloud's resiliency and on-demand access for a robust and green solution. Interestingly, most of the cloud vendors provide APIs to access cloud resources in an efficient manner different than how an MPI implementation would avail of those resources. We have done extensive research to identify the pain-points for designing and implementing an MPIlike framework for cloud platforms. Our research has provided us with vital guidelines that we are sharing in this paper. We present the details of the key components required for such a framework along with our experience while implementing a preliminary MPI-like framework over Azure dubbed cloudMPI and evaluate its pros and cons. A large GIS application has been ported over cloudMPI to study its effectiveness and limitations. 1

The Dynamic Computational Model and the New Era of Cloud Computation Using Microsoft Azure

“…It monitors the allocation of resources, monitors the health of computing and storage services. Here fabric controller will look into the failure recovery also [6].…”

Using the Cloud for Parameter Estimation Problems: Comparing Spark vs MPI with a Case-Study

“…Additionally, in an attempt of converging cloud platforms and HPC, projects like DataMPI [16] or CloudMPI [17] arose. DataMPI [16] aimed at extending MPI by keyvalue pair based communication operations to provide high performance communications in cloud scenarios.…”

“…DataMPI [16] aimed at extending MPI by keyvalue pair based communication operations to provide high performance communications in cloud scenarios. The cloudMPI [17] framework aimed at designing and implementing an MPI-like framework for cloud platforms, having an early implementation on the Azure cloud platform. Unfortunately, none of these projects seems to be active at this moment.…”

DEF - a programming language agnostic framework and execution environment for the parallel execution of library routines

“…This means that there are examples for Cloud based MPI implementations, e.g. [19,20]. And there are also few publications that deal with distributed shared memory solutions in a Cloud environment.…”

“…This means that the VM images for the Cloud Controller and for the worker must be adapted for the different Cloud infrastructure environments (e.g. AWS, Azure 20 , OpenStack 21 , Eucalyptus). The initialization of the cluster, implemented in init_cluster(), will be based on the following scenario:…”