Predicting cloud performance for HPC applications before deployment

Mariani, Giovanni; Anghel, Andreea; Jongerius, Rik; Dittmann, Gero

doi:10.1016/j.future.2017.10.048

Cited by 17 publications

(26 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…COMPASS is an example of tool that demonstrates the feasibility of creating precise performance models of parallel programs for calculating QoS and cost parameters of computation and platform components of HPC Shelf. Mariani et al employ machine learning to predict the performance for HPC applications running in clouds [Mariani et al 2018]. Also, Cunha et al have used machine learning to help the placement of jobs in clouds [Cunha et al 2017].…”

Section: Related Work and Contributionsmentioning

confidence: 99%

Contextual Contracts for Component-Based Resource Abstraction in a Cloud of HPC Services

Alam¹,

Carvalho²

2019

Anais Do XX Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2019)

View full text Add to dashboard Cite

The efforts to make cloud computing suitable for the requirements of HPC applications have motivated us to design HPC Shelf, a cloud computing platform of services for building and deploying parallel computing systems for large-scale parallel processing. We introduce Alite, the system of contextual contracts of HPC Shelf, aimed at selecting component implementations according to requirements of applications, features of targeting parallel computing platforms (e.g. clusters), QoS (Quality-of-Service) properties and cost restrictions. It is evaluated through a small-scale case study employing a componentbased framework for matrix-multiplication based on the BLAS library.

show abstract

Section: Related Work and Contributionsmentioning

confidence: 99%

Contextual Contracts for Component-Based Resource Abstraction in a Cloud of HPC Services

Alam¹,

Carvalho²

2019

Anais Do XX Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2019)

View full text Add to dashboard Cite

show abstract

“…Mariani e outros [12] apresentaram uma terceira abordagem de predição. A técnica consiste em criar um modelo da infraestrutura e um modelo independente de hardware da aplicação e utilizar ambos como entrada para um algoritimo Random Forest para determinar a melhor infraestrutura.…”

Section: Trabalhos Relacionadosunclassified

“…Definir qual configuração computacional apresenta o melhor custo-benefício nãoé uma tarefa fácil, pois para isso deve-se saber de antemão a quantidade de recursos que uma aplicação necessita para atingir seu pico de desempenho [4]. Mesmo possuindo plena compreensão sobre as aplicações, o usuário pode não dispor da percepção de alguns detalhes de funcionamento da nuvem computacional -percepção da interação de sua aplicação com as camadas de abstração da nuvem, detalhes do hardware que executa a máquina virtual na nuvem ou consequências de compartilhar o ambiente da nuvem com outros usuários [11,10].…”

Section: Introductionunclassified

Aplicação da técnica Paramount Iteration nas aplicações BLAST e DNN-ROM na nuvem computacional

Tavares

Reis

Brunetta

et al. 2019

Anais Do XX Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2019)

View full text Add to dashboard Cite

O crescimento da tendência da computação em nuvem traz novos desafios para a comunidade de computação de alto desempenho. Por possuir um amplo número de recursos, predizer a melhor configuração para uma aplicação especı́fica é uma tarefa custosa e de alto consumo de tempo e principalmente financeiro. A técnica paramount iteration consiste em executar uma parcela da aplicação a fim de determinar o comportamento esperado neste ambiente computacional quando executado por completo. Este artigo valida e utiliza a técnica paramount iteration para as aplicações BLAST e DNN-ROM, sendo possı́vel determinar o melhor ambiente de computação em nuvem para estas.

show abstract

“…Similarly, the existence of variety is determined using heterogeneity ratio and predefined thresholds. With the large and complex datasets, one time profiling seems infeasible [30], [33] for computing the memory consumption and time. Similarly, static complexity analysis is not a viable and precise approach for real-world application codes due to nondeterministic execution paths [30], [34], [35].…”

Section: Introductionmentioning

confidence: 99%

“…Similarly, static complexity analysis is not a viable and precise approach for real-world application codes due to nondeterministic execution paths [30], [34], [35]. In this manner, the proposed prediction based approach using machine learning and instrumentation appears to be reasonable for big data applications [33].…”

Section: Introductionmentioning

confidence: 99%

Towards a Novel Framework for Automatic Big Data Detection

Ahmed

Ismail

2020

IEEE Access

View full text Add to dashboard Cite

Big data is a "relative" concept. It is the combination of data, application, and platform properties. Recently, big data specific technologies have emerged, including software frameworks, databases, hardware accelerators, storage technologies, etc. However, the automatic selection of these solutions for big data computations remains a non-trivial task. Presently, the big data tools are selected by analyzing the problem manually, or by using several performance prediction techniques. The manual identification is based on the data properties only, whereas the performance predictors only estimate basic execution metrics without linking them with big data (3Vs) thresholds. Hence, both ways of identification are mostly incorrect, which can lead to inefficient use of 3Vs optimizations, resulting into global inefficiency, reduced system performance, increasing power consumption, requiring greater effort on the part of the programming team, and misallocation of the hardware resources required for the task. In this regard, a novel framework has been proposed for automatic detection of 3Vs (Volume, Velocity, Variety) of big data, using machine learning. The detection is done through static code features, data, and platform properties, leading to relevant tool selection, and code generation, with minimal overheads, lesser programmer interventions, higher usability, and portability. Instead of handling each application with big data specialized solutions, or manually identifying the 3Vs, the framework can automatically detect and link the 3Vs to the relevant optimizations. Several standard applications have been tested using the proposed framework. In the case of volume, the average detection accuracy is up to 97.8% for seen and 95.9% for unseen applications. In the case of velocity, the average detection accuracy is up to 97.3% for seen and 92.6% for unseen applications. There is no margin of error in variety detection, as it has straightforward computations without any predictions. Furthermore, an airline recommendation system case study strengthens the effectiveness of the proposed approach.

show abstract

Predicting cloud performance for HPC applications before deployment

Cited by 17 publications

References 20 publications

Contextual Contracts for Component-Based Resource Abstraction in a Cloud of HPC Services

Contextual Contracts for Component-Based Resource Abstraction in a Cloud of HPC Services

Aplicação da técnica Paramount Iteration nas aplicações BLAST e DNN-ROM na nuvem computacional

Towards a Novel Framework for Automatic Big Data Detection

Contact Info

Product

Resources

About