A Framework for Elastic Execution of Existing MPI Programs

Abstract

Abstract-There is a clear trend towards using cloud resources in the scientific or the HPC community, with a key attraction of cloud being the elasticity it offers. In executing HPC applications on a cloud environment, it will clearly be desirable to exploit elasticity of cloud environments, and increase or decrease the number of instances an application is executed on during the execution of the application, to meet time and/or cost constraints. Unfortunately, HPC applications have almost always been designed to use a fixed number of resources.This paper describes our initial work towards the goal of making existing MPI applications elastic for a cloud framework. Considering the limitations of the MPI implementations currently available, we support adaptation by terminating one execution and restarting a new program on a different number of instances. The components of our envisioned system include a decision layer which considers time and cost constraints, a framework for modifying MPI programs, and a cloud-based runtime support that can enable redistributing of saved data, and support automated resource allocation and application restart on a different number of nodes.Using two MPI applications, we demonstrate the feasibility of our approach, and show that outputting, redistributing, and reading back data can be a reasonable approach for making existing MPI applications elastic.

DEF - a programming language agnostic framework and execution environment for the parallel execution of library routines

“…This means that there are examples for Cloud based MPI implementations, e.g. [19,20]. And there are also few publications that deal with distributed shared memory solutions in a Cloud environment.…”

“…We therefore chose Linux as operating system and all installed runtime environments are also free of charge. Even the MATLAB runtime 19 can be installed free of license costs on each worker, which allows to also execute (compiled) library routines written in MATLAB to be executed in parallel on the DEF workers.…”

Elasticidade em cloud computing: conceito, estado da arte e novos desafios

“…Ela pode representar o emprego de uma interface de programação (API) oferecida pelo middleware de nuvem para que o programador escreva o seu próprio gerenciador ou o uso de ferramentas, sejam elas em linha de comando ou gráficas. Quanto às abordagens automáticas, a reativa é marcada pelo emprego do mecanismo regra-condição-ação [19,20,21]. Nesse caso, é comum o emprego de thresholds na escrita das regras para gerência das métricas.…”

“…Tomando como métrica o uso de CPU, outros itens também podem ser levados em conta, como o relógio do processador, o tempo de execução de tarefas (makespan), o número de instruções e o redimensionamento de prioridades de uso de CPU. Por exemplo, Raveendran et al [21] trabalham com um tempo máximo para a conclusão de cada uma das fases de um programa iterativo. No início de uma nova iteração, busca-se verificar se o tempo da anterior não excedeu um threshold, e, em caso positivo, uma nova máquina virtual é criada.…”

On exploring proactive cloud elasticity for internet of things demands

“…However, the state-of-the-art does not present satisfactory solutions in which this kind of middlewares are capable of adapt themselves according to processing demands. Embora existam benefícios para sistemas IoT, a elasticidade em nuvem é mais largamente explorada em arquiteturas cliente-servidor, como vídeo sob-demanda, lojas online, aplicações BOINC (Berkeley Open Infrastructure for Network Computing) [8], governança eletrônica e Web Services [9]. Apesar de diversos estudos na direção de criar middlewares RFID compatíveis com o padrão EPCglobal, ainda há uma lacuna no que diz respeito à elasticidade de recursos para tais sistemas.…”