Virtual Clusters in the Jetstream Cloud

Coulter, J. Eric; Abeysinghe, Eroma; Pamidighantam, Sudhakar; Pierce, Marlon

doi:10.1145/3355738.3355752

“…• There is negligible chance that the orchestration layer will have control of all intermediate network components across all participating cloud sites, 13 INDIGO Virtual Router: https://github.com/indigo-dc/ansible-role-indigovr therefore an overlay network is required to provide for a private connection among geographically distant sites.…”

Section: Design Assumptionsmentioning

confidence: 99%

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

Caballer,

Antonacci,

Šustr

et al. 2021

Preprint

0

View full text Add to dashboard Cite

Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network.

show abstract

“…In many cases, researchers do not have access to local HPC hardware on which to test and profile their software, before moving to large-scale XSEDE environments such as Comet or Stampede2. In order to provide an HPC testbed environment to such researchers, the XCRI team has developed a selfscaling virtual cluster [23] for use on Openstack clouds, such as Jetstream [7]. This toolkit provides researchers with a basic HPC system, based on the OpenHPC [13] project using the same software as those powering the systems available within XSEDE.…”

Section: B the Need For Cloud-integrated Computational Softwarementioning

confidence: 99%

“…Nonetheless, over the course of providing virtual HPC environments for dozens of projects [23], it has become quite clear that the administrative overhead of maintaining scientific software does not scale well within a small team, nor does it effectively empower users to easily transfer software and workflows into XSEDE without assistance. While the premise of familiarizing users with an HPC environment holds in situations where researchers are new to HPC, things fall apart in the case of established research teams who want to take customized software from a cloud environment with root access into a heavily managed multi-user system.…”

Section: B the Need For Cloud-integrated Computational Softwarementioning

confidence: 99%

“…Otherwise, the addition of scientific software can be done simply by pulling down a Singularity image, and jobs can be run without further intervention. A more detailed discussion of the configuration of any individual VC is provided in [23]. For clarity in the following discussion in Section V-A, it is worth mentioning that the software on the cluster is installed from repositories provided by the OpenHPC [13] project, including Singularity version 3.4.1 and OpenMPI version 3.1.0 at the time of writing.…”

Section: Deployment Processmentioning

confidence: 99%

“…We have additionally demonstrated a simple, flexible HPCstyle infrastructure which is deployable across multiple cloud providers at nearly the push of a button, which can provide an invaluable testing or bursting environment for researchers lacking in local resources. Additionally, this infrastructure is already battle-tested behind the scenes of numerous Science Gateways [23], [61]- [63].…”

Section: Practical Outcomesmentioning

confidence: 99%

See 2 more Smart Citations

Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing

Vaillancourt¹,

Coulter²,

Knepper³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Container technologies such as Docker have become a crucial component of many software industry practices especially those pertaining to reproducibility and portability. The containerization philosophy has influenced the scientific computing community, which has begun to adopt -and even develop -container technologies (such as Singularity). Leveraging containers for scientific software often poses challenges distinct from those encountered in industry, and requires different methodologies. This is especially true for HPC. With an increasing number of options for HPC in the cloud (including NSF-funded cloud projects), there is strong motivation to seek solutions that provide flexibility to develop and deploy scientific software on a variety of computational infrastructures in a portable and reproducible way. The flexibility offered by cloud services enables virtual HPC clusters that scale on-demand, and the Cyberinfrastructure Resource Integration team in the XSEDE project has developed a set of tools which provides scalable infrastructure in the cloud. We now present a solution which uses the Nix package manager in an MPI-capable Docker container that is converted to Singularity. It provides consistent installations, dependencies, and environments in each image that are reproducible and portable across scientific computing infrastructures. We demonstrate the utility of these containers with cluster benchmark runs in a self-scaling virtual cluster using the Slurm scheduler deployed in the Jetstream and Aristotle Red Cloud OpenStack clouds. We conclude this technique is useful as a template for scientific software application containers to be used in the XSEDE compute environment, other Singularity HPC environments, and cloud computing environments.

show abstract

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

Caballer

¹

,

Antonacci

²

,

Šustr

³

et al. 2021

View full text Add to dashboard Cite

Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network.

show abstract

Virtual Clusters in the Jetstream Cloud

Cited by 10 publications

References 8 publications

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing

Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

Contact Info

Product

Resources

About