Containers in HPC: A Scalability and Portability Study in Production Biological Simulations

Rudyy, Oleksandr; García-Gasulla, Marta; Mantovani, Filippo; Santiago, Alfonso; Sirvent, Raül; Vázquez, Mariano

doi:10.1109/ipdps.2019.00066

Cited by 31 publications

(13 citation statements)

References 18 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [47], authors evaluated the network overhead of Docker and Singularity containers using HPCG and miniFE workloads. Rudyy et al [48] also compared three different container technologies on computational fluid dynamics (CFD) and concluded Singularity is the suitable container solution for HPC workloads that provide the same execution time as Bare-Metal. Mavridis and Karatza [49] studied containers on VMs configuration.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms

Ghatrehsamani

Denninnart

Bacik

et al. 2020

49th International Conference on Parallel Processing - ICPP

View full text Add to dashboard Cite

Cloud providers offer a variety of execution platforms in form of bare-metal, VM, and containers. However, due to the pros and cons of each execution platform, choosing the appropriate platform for a specific cloud-based application has become a challenge for solution architects. The possibility to combine these platforms (e.g., deploying containers within VMs) offers new capacities that makes the challenge even further complicated. However, there is a little study in the literature on the pros and cons of deploying different application types on various execution platforms. In particular, evaluation of diverse hardware configurations and different CPU provisioning methods, such as CPU pinning, have not been sufficiently studied in the literature. In this work, the performance overhead of container, VM, and bare-metal execution platforms are measured and analyzed for four categories of real-world applications, namely video processing, parallel processing (MPI), web processing, and No-SQL, respectively representing CPU intensive, parallel processing, and two IO intensive processes. Our analyses reveal a set of interesting and sometimes counterintuitive findings that can be used as best practices by the solution architects to efficiently deploy cloud-based applications. Here are some notable mentions: (A) Under specific circumstances, containers can impose a higher overhead than VMs; (B) Containers on top of VMs can mitigate the overhead of VMs for certain applications; (C) Containers with a large number of cores impose a lower overhead than those with a few cores.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Several research works compared containers versus VMs and Bare-Metal [2]- [5], [42], [46]- [48]. Only few of these studies explored VMCN configuration, which is a popular platform in the cloud [49].…”

Section: Related Workmentioning

confidence: 99%

The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms

Ghatrehsamani

Denninnart

Bacik

et al. 2020

49th International Conference on Parallel Processing - ICPP

View full text Add to dashboard Cite

show abstract

“…On the front of linear algebra library, we are in contact with the developer of the Performance Library in Arm, and we plan to continue our fruitful collaboration for improving the performance of TensorFlow on Arm platforms. As a contribution to the data centers, following our previous experience described in [21], we plan to evaluate TensorFlow in combination with containers technologies for an easier deployment for the users.…”

Section: Related and Future Workmentioning

confidence: 99%

TensorFlow on State-of-the-Art HPC Clusters: A Machine Learning use Case

Ramirez-Gargallo

García-Gasulla

Mantovani

2019

2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

Self Cite

View full text Add to dashboard Cite

The recent rapid growth of the data-flow programming paradigm enabled the development of specific architectures, e.g., for machine learning. The most known example is the Tensor Processing Unit (TPU) by Google. Standard data-centers, however, still can not foresee large partitions dedicated to machine learning specific architectures. Within data-centers, the High-Performance Computing (HPC) clusters are highly parallel machines targeting a broad class of compute-intensive workflows, as such they can be used for tackling machine learning challenges. On top of this, HPC architectures are rapidly changing, including accelerators and instruction sets other than the classical x86 CPUs. In this blurry scenario, identifying which are the best hardware/software configurations to efficiently support machine learning workloads on HPC clusters is not trivial. In this paper, we considered the workflow of TensorFlow for image recognition. We highlight the strong dependency of the performance in the training phase on the availability of arithmetic libraries optimized for the underlying architecture. Following the example of Intel leveraging the MKL libraries for improving the TensorFlow performance, we plugged the Arm Performance Libraries into TensorFlow and tested on an HPC cluster based on Marvell ThunderX2 CPUs. Also, we performed a scalability study on three state-of-the-art HPC clusters based on different CPU architectures, x86 Intel Skylake, Arm-v8 Marvell ThunderX2, and PowerPC IBM Power9.

show abstract

“…As an example, the article Rudyy et al [123] analyses the performance overhead induced by the use of three different container technologies (Docker, Singularity and Shifter) by comparing it with native execution. Another example is the work done by Hu, Zhang, and Chen [61] where the CPU performance, memory and network bandwidth between Singularity and bare metal are evaluated in detail by means of different benchmarks.…”

Section: Containerized Computingmentioning

confidence: 99%

Serverless Computing Strategies on Cloud Platforms

Delgado¹,

María²

View full text Add to dashboard Cite

First of all I want to thank my parents for always being there for me and supporting me in all the decisions I have made throughout my life. To my sister, who is my role model, for encouraging me to embark on this completely new adventure for me. To my husband for always being by my side and making my days happy just by his presence. I would like to thank all my GRyCAP colleagues for their support throughout this thesis and my career in the research group. Especially all my gratitude to Ignacio Blanquer and Germán Moltó for trusting me from the first day and providing me with all the tools and help necessary for the development of this research. With all of you I will be eternally grateful. I want to express all my gratitude to all my friends who in one way or another have always supported and encouraged me in difficult moments. I would also like to thank all the people who have participated in my training as a person. Thanks to all of them I have been advancing in my life and I have reached this point. iii

show abstract

Containers in HPC: A Scalability and Portability Study in Production Biological Simulations

Cited by 31 publications

References 18 publications

The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms

The Art of CPU-Pinning: Evaluating and Improving the Performance of Virtualization and Containerization Platforms

TensorFlow on State-of-the-Art HPC Clusters: A Machine Learning use Case

Serverless Computing Strategies on Cloud Platforms

Contact Info

Product

Resources

About