Democratization of HPC cloud services with automated parallel solvers and application containers

Muhtaroglu, Nitel; Arı, İsmail; Kolcu, Birkan

doi:10.1002/cpe.4782

Cited by 4 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Muhtaroglu et al investigated several design choices for HPC services at different layers of the cloud computing architecture to simplify and broaden its use cases. They compared direct versus iterative parallel linear equation solvers for the platform‐as‐a‐service layer.…”

Section: Special Issue Papersmentioning

confidence: 99%

Special issue on High performance computing conference (BASARIM‐2017)

Unat

Aktaş

2018

Concurrency and Computation

View full text Add to dashboard Cite

Section: Special Issue Papersmentioning

confidence: 99%

Special issue on High performance computing conference (BASARIM‐2017)

Unat

Aktaş

2018

Concurrency and Computation

View full text Add to dashboard Cite

“…To resolve this problem, many scholars have begun to explore automatic parallelization platforms that can automatically transform serial programs into parallel programs. At present, there have been many achievements, but mainly for specific research directions and areas 14‐16 …”

Section: Introductionmentioning

confidence: 99%

“…At present, there have been many achievements, but mainly for specific research directions and areas. [14][15][16] For this reason, the Par4All automatic parallel software package was designed and developed by the HPC Project Lab in France, which has undoubtedly brought automatic parallel code generation technology to a new level of practicality. Par4All is an open-source automatic parallel compilation and optimization module that can automatically generate OpenMP/CUDA/OpenCL parallel code files in the C or Fortran languages.…”

mentioning

confidence: 99%

Preliminary study on the automatic parallelism optimization model for image enhancement algorithms based on Intel's® Xeon Phi

Huang

Yang

Jian

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

In unmanned aerial vehicle (UAV) image-processing applications, one needs to implement different parallel image-enhancement algorithms on several high-performance computing platforms utilizing various programming models. To speed up the parallelization procedure and improve its efficiency, the automatic parallel software package, Par4All, is applied in this work. We find that the performance of the original automatic parallelization algorithm produced with Par4All is inefficient. To resolve this problem, we propose different optimization approaches for Par4All based on Intel®'s Xeon Phi high-performance computing platform that are based on the structural features of the image-enhancement algorithms, which can further optimize the original parallel algorithm. These approaches mainly include: (1) Par4All automatic parallel search module optimization, (2) dynamic thread setting optimization, and (3) the collaborative parallelization of both CPU and many integrated core (MIC) processors.According to the results of the comparison experiments involving different algorithms, it is shown that the proposed optimization approaches for these kinds of algorithms can significantly improve the performance of automatic parallel algorithms. The acceleration ratio increases approximately by 30%, 70%, and 80% for the multiscale Retinex, Gaussian-filtering and median-filtering algorithms, respectively. As continuation and deepening of our previous research work, this research has the potential to be beneficial for other researchers in image-processing applications with image-enhancement algorithms. K E Y W O R D Sautomatic parallelism, image-enhancement algorithms, Par4All, Intel® Xeon Phi, unmanned aerial vehicles INTRODUCTIONAs mentioned by References 1, it is known that some image-enhancement algorithms, for example, median filtering, Gaussian filtering, wavelet transform, and the multiscale Retinex (MSR) algorithm, are used when preprocessing large amounts of unmanned aerial vehicle (UAV) remote-sensing (RS) images to solve the problems of poor clarity, insufficient contrast and low adaptability. In addition, many-cores computing platforms, e.g., graphics processing units (GPUs) and many integrate cores (MICs), have been involved in the development of some parallel algorithms to speed up the processing times 2 ; such approaches are expected to become mainstream in the near future. Experimental results have shown that parallelized algorithms can significantly improve computation speed and provide better acceleration efficiency. [3][4][5] Usually, there are different kinds of serial

show abstract

MaLeFICE: Machine learning support for continuous performance improvement in computational engineering

Sonmezer

Muhtaroglu

Arı

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Computer aided engineering (CAE) practices improved drastically within the last decade due to ease of access to computing resources and open-source software. However, increasing complexity of hardware and software settings and the scarcity of multiskilled personnel rendered the practice inefficient and infeasible again. In this article, we present a method for continuous performance improvement in computational engineering that combines online performance profiling with machine learning (ML). To test the viability of this method, we provide a detailed analysis for solution time estimation of finite element analysis (FEA) jobs based on multidimensional models. These models combine numerous matrix features (matrix size, density, bandwidth, etc.), solver features (direct-iterative, preconditioning, tolerance), and hardware features (core count, virtual-physical). We repeat our analysis over different machines as well as docker containers to demonstrate applicability over different platforms. Next, we train supervised and unsupervised ML algorithms over commonly used, realistic FEA benchmarks and compare accuracy of different models. Finally, we design two new ML-based online batch schedulers called shortest predicted time first (SPTF) and shortest cluster time first (SCTF), which are comparable in performance to the optimal, but offline shortest job first (SJF) scheduler. We find that ML-based profiling and scheduling can reduce the average turnaround times by 2×-5× over other alternatives.

show abstract

Democratization of HPC cloud services with automated parallel solvers and application containers

Cited by 4 publications

References 27 publications

Special issue on High performance computing conference (BASARIM‐2017)

Special issue on High performance computing conference (BASARIM‐2017)

Preliminary study on the automatic parallelism optimization model for image enhancement algorithms based on Intel's® Xeon Phi

MaLeFICE: Machine learning support for continuous performance improvement in computational engineering

Contact Info

Product

Resources

About