Jupyter as Common Technology Platform for Interactive HPC Services

Milligan, Michael

doi:10.1145/3219104.3219162

Cited by 10 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, the hub manager governs the intra-and inter-federation (i.e., eduGAIN [34]) metadata of SAML entities (i.e., SAML IdPs or SPs), and maintains the history of user events, such as consent records; • OTS/STO (OIDC to SAML/SAML to OIDC) Broker: Many science applications, such as Jupyter [35] and Globus [36], are beginning to support the OIDC protocol.…”

Section: Federated-access Management Systemmentioning

confidence: 99%

Federated-Access Management System and Videoconferencing Applications: Results from a Pilot Service during COVID-19 Pandemic

Chae²,

Jang

et al. 2021

Electronics

View full text Add to dashboard Cite

Videoconferencing has become a crucial enabler for sustainable collaboration and learning during the COVID-19 pandemic. However, national regulations often restrict public institutions from introducing commercial videoconferencing services. Open-source software is an attractive option for institutions if it can be protected from potential security threats while ensuring high usability. Unfortunately, to the best of our knowledge, we hardly find available open-source videoconferencing applications in the literature that stress their usability and adopt security-related frameworks. This study presents a federated-access management system called trustHub, which was developed to enable flexible and elaborate access control and protocol-agnostic user authentication. In addition, we introduce two videoconferencing applications that aim to improve the usability of leveraged open-source software. They are prototyped to operate in concert with trustHub to take firm access control and accept various types of identity providers. Consequently, using data collected from trustHub and a prototyped videoconferencing application over a 10-month period, we conduct a comprehensive analysis to understand the usage patterns of federated access and videoconferencing during the pandemic and, thus, verify their feasibility indirectly.

show abstract

Section: Federated-access Management Systemmentioning

confidence: 99%

Federated-Access Management System and Videoconferencing Applications: Results from a Pilot Service during COVID-19 Pandemic

Chae²,

Jang

et al. 2021

Electronics

View full text Add to dashboard Cite

show abstract

“…Appendix shows the advantages of Spack compared to other software installation methods. If desired, it is then possible to deploy HPC‐as‐a‐Service for other users on top of such a custom HPC environment, using auxiliary AWS functionalities (Amazon, 2019f, 2019a, 2019c) and multiuser serving frameworks like JupyterHub (Glick & MacHe, 2019; Milligan, 2018; Prout et al, 2017; Sarajlic et al, 2018); this can be useful for classes and workshops where the customization of software environment is not important.…”

Section: Hpc Workflow On the Aws Cloudmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

“…There are numerous efforts at various DOE and NSF facilities to enable Jupyter in HPC. Work done on Batchspawner [25] at the University of Minnesota has enabled a lot of the initial batch system integration in JupyterHub, and is being used as the core to build a lot of the advanced resource scheduling infrastructure at NERSC. Tools like Parsl [26], Dask [4], Spark, and IPyParallel [27] manage backend workers that can be used to facilitate farming out tasks to a distributed set of nodes.…”

Section: Related Workmentioning

confidence: 99%

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

Cholia

Heagy

Henderson

et al. 2020

2020 IEEE/ACM HPC for Urgent Decision Making (UrgentHPC)

View full text Add to dashboard Cite

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case -applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow -reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-time streams of HDF5 data.

show abstract

Jupyter as Common Technology Platform for Interactive HPC Services

Cited by 10 publications

References 10 publications

Federated-Access Management System and Videoconferencing Applications: Results from a Pilot Service during COVID-19 Pandemic

Federated-Access Management System and Videoconferencing Applications: Results from a Pilot Service during COVID-19 Pandemic

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

Contact Info

Product

Resources

About