Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study

Abstract

The High-Performance Computing (HPC) community has recently started to use containerization to obtain fast, customized, portable, flexible, and reproducible deployments of their workloads. Previous work showed that deploying an HPC workload into a single container can keep bare-metal performance. However, there is a lack of research on multi-container deployments that partition the processes belonging to each application into different containers. Partitioning HPC applications have shown to improve their performance on virtual machines by allowing them to be set affinity to a NUMA (Non-Uniform Memory Access) domain. Consequently, it is essential to understand the performance implications of distinct multi-container deployment schemes for HPC workloads, focusing on the impact of the container granularity and its combination with processor and memory affinity. This paper presents a systematic performance comparison and analysis of multi-container deployment schemes for HPC workloads on a single-node platform, which considers different containerization technologies (including Docker and Singularity), two different platform architectures (UMA and NUMA), and two application subscription modes (exactly-subscription and over-subscription). Our results indicate that finer-grained multi-container deployments, on one side, can benefit the performance of some applications with low inter-process communication, especially in over-subscribed scenarios and when combined with affinity but, on the other side, they can incur some performance degradation for communicationintensive applications when using containerization technologies that deploy isolated network namespaces.

Container Technologies for ARM Architecture: A Comprehensive Survey of the State-of-the-Art

“…The continuous involvement of containers in HPC created a new research direction of multi-container deployment schemes for HPC. Peini et al [237] compared and analyzed multi-container development schemes based on Docker and Singularity on two different platforms: UMA and NUMA and showed that Docker and Singularity container technology has performance overhead for MPI when running on multiple containers. Research work [238] used Kubernetes on HPC and compared the performance against Docker Swarm and bare metal.…”

Fine-Grained Scheduling for Containerized HPC Workloads in Kubernetes Clusters

“…Moreover, they provided a rule to decide the number of containers per pod by considering the characteristics of the application. Our previous papers [12] [13] demonstrated through standalone executions that some types of containerized HPC applications achieve better performance when exploiting multi-container deployments which partition the processes that belong to each application into multiple containers in each node, and when constraining each of those containers to a single NUMA domain or pinning them to specific processors. These works show some ways to achieve better performance for HPC workloads in the Cloud, but those insights have not yet been integrated and utilized by the current Cloud orchestrators.…”

“…Our previous systematical performance studies [12] [13] have demonstrated through standalone executions that some types of containerized HPC applications achieve better performance when exploiting multi-container deployments which partition the processes that belong to each application into multiple containers in each node and when constraining each of those containers to a single NUMA (Non-Uniform Memory Access) domain or pinning them to specific processors. However, these deployment schemes have not yet been integrated in multiprogrammed environments for HPC workloads by current Cloud orchestrators.…”

Performance characterization of containerization for HPC workloads on InfiniBand clusters: an empirical study

“…In our previous work [16], we enabled the interconnection across hosts through TCP/IP protocol between Singularity instances running Big Data applications. Moreover, in our latest work [3], we performed a performance analysis of multi-container deployments with different container granularity for a number of HPC applications, but only with a single host and the TCP/IP protocol.…”

“…Following the trend of Cloud computing, the HPC community has also started to adopt containerization instead of hardware virtualization to benefit from some of its wellknown advantages [1], such as the encapsulation of specific software environments for each user, which allows for customization, portability, and research reproducibility; the isolation of users from the underlying system and from other users, which allows for security and fault protection; and the agile and fine-grain resource allocation and balancing, which allows for efficient cluster utilization and failure recovery [2,3].…”