Switching Software in Science: Motivations, Challenges, and Solutions

Wessel, Jan R.; Gorgolewski, Krzysztof J.; Bellec, Pierre

doi:10.1016/j.tics.2019.01.004

“…In a similar vein, data scientists frequently carry out and communicate computations in computational notebook settings, such as Jupyter Notebooks (Kluyver et al 2016) or R Markdown documents from within RStudio (Allaire et al 2019;Xie 2018). This is also true within academia, as research moves towards general, open-source tools (Voytek 2017;Wessel et al 2019;Wilson et al 2017). We encourage that course design should prioritize the use of these tools as students with experience in common languages and tools are immediately more employable and more competent at a variety of technical tasks.…”

Section: Prioritizing Relevant Toolingmentioning

confidence: 94%

Teaching Creative and Practical Data Science at Scale

Donoghue

¹

,

Voytek

²

,

Ellis

³

2021

Journal of Statistics and Data Science Education

View full text Add to dashboard Cite

“…Online cloud‐based CI services such as TravisCI (https://travis-ci.com/) and Azure pipelines (Microsoft, 2019) automatically build the software source code and run the unit tests at every code commit and report any potential errors during the build or test stage. CI is a standard practice in software engineering (Duvall et al, 2007) and is also useful for scientific data analysis (Wessel et al, 2019) and HPC application development (Sampedro et al, 2018). Without CI, software developers would need to manually find and track software issues.…”

Section: Benefits Of Cloud Computing For Earth Science Researchmentioning

confidence: 99%

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

¹

,

Jacob

²

,

Lin

³

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

“…Online cloud-based Continuous Integration (CI) services such as TravisCI (https://travis-ci.com/) and Azure pipelines (Microsoft, 2019) automatically build the software source code and run the unit tests at every code commit, and report any potential errors during the build or test stage. CI is a standard practice in software engineering (Duvall et al, 2007), and is also useful for scientific data analysis (Wessel et al, 2019) and HPC application development (Sampedro et al, 2018). Without CI, software developers would need to spend long time manually finding and tracking software issues.…”

Section: Benefits Of Cloud Computing For Earth Science Researchmentioning

confidence: 99%

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Zhuang¹,

Jacob²,

Lin³

et al. 2020

Preprint

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input datasets. However, slow inter-node communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services (AWS) cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface (MPI) library configuration and inter-node communication protocol are critical to this success. Application to the GEOS-Chem model of atmospheric chemistry at global 50 km horizontal resolution shows efficient scaling up to at least 1152 cores, with performance and cost comparable to the NASA Pleiades supercomputing cluster.

show abstract

Switching Software in Science: Motivations, Challenges, and Solutions

Cited by 7 publications

References 2 publications

Teaching Creative and Practical Data Science at Scale

Teaching Creative and Practical Data Science at Scale

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Enabling high-performance cloud computing for Earth science modeling on over a thousand cores: application to the GEOS-Chem atmospheric chemistry model

Contact Info

Product

Resources

About