Using Jupyter for Reproducible Scientific Workflows

Beg, Marijan; Taka, Juliette; Kluyver, Thomas; Konovalov, Alexander; Ragan-Kelley, Min; Thiéry, Nicolas M.; Fangohr, Hans

doi:10.1109/mcse.2021.3052101

Cited by 73 publications

(53 citation statements)

References 11 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this shows the potential for combining exploration and explanation of new analysis methods in a common framework, Rule et al explain how reproducibility is better achieved when users develop their notebooks following general purpose guidelines [13]. Eventually, to really reproduce the analysis, users have be familiar with additional systems, (i.e., data and software repositories) [1], requiring them to learn new technologies and formats. Obtaining actionable metadata that would also foster the automation of…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SWIRRL. Managing Provenance-aware and Reproducible Workspaces

et al. 2022

View full text Add to dashboard Cite

Modern interactive tools for data analysis and visualisation are designed to expose their functionalities as a service through the Web. We present in this paper a Web API (SWIRRL) that allows Virtual Research Environments (VREs) to easily integrate such tools in their websites and re-purpose them to their users. The API deals, on behalf of the clients, with the underlying complexity of allocating and managing resources within a target cloud platform. By combining storage and containerised services, offering analysis notebooks and other visualisation software, the API creates dedicated working sessions on-demand, which can be accessed collaboratively. Thanks to the API’s support for workflow execution, SWIRRL workspaces can be automatically populated with data of interest collected from external data providers. The system keeps track of updates and changes affecting the data and the tools by adopting versioning and standard provenance technologies. Users are provided with interactive controls enabling traceability and recovery actions, including the possibility of creating executable snapshots of their environments. SWIRRL is built in cooperation with two research infrastructures in the field of solid earth science and climate data modeling. We report on the particular adoptions and use cases.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Pursuing reproducible research in Jupyter Notebook requires users to adopt general good practices [13] and learn how to use complementary systems [1]. We illustrate how SWIRRL assists users in this objective.…”

Section: Automating Notebook Reproducibilitymentioning

confidence: 99%

SWIRRL. Managing Provenance-aware and Reproducible Workspaces

et al. 2022

View full text Add to dashboard Cite

show abstract

“…For data-proximate computational resources, we have implemented a JupyterHub, an open-source platform gaining rapid attention in the scientific community (Fangohr et al, 2019;Beg et al, 2021), on Google Cloud Platform in collaboration with 2i2c.org, a non-profit organization based in the U.S. The authentication for each user/collaborator to the JupyterHub is done via Github.…”

Section: Cloud-based Jupyterhubmentioning

confidence: 99%

Cloud-based framework for inter-comparing submesoscale permitting realistic ocean models

Uchida

Sommer

Stern

et al. 2022

Preprint

View full text Add to dashboard Cite

Abstract. With the increase in computational power, ocean models with kilometer-scale resolution have emerged over the last decade. These models have been used for quantifying the energetic exchanges between spatial scales, informing the design of eddy parametrizations and preparing observing networks. The increase in resolution, however, has drastically increased the size of model outputs, making it difficult to transfer and analyze the data. Nonetheless, it is of primary importance to assess more systematically the realism of these models. Here, we showcase a cloud-based analysis framework proposed by the Pangeo Project that aims to tackle such distribution and analysis challenges. We analyze the output of eight submesoscale-permitting simulations, all on the cloud, for a crossover region of the upcoming Surface Water and Ocean Topography (SWOT) altimeter mission near the Gulf Stream separation. The models used in this study are run with the NEMO, CROCO, MITgcm, HYCOM, FESOM and FIO-COM code bases. The cloud-based analysis framework: i) minimizes the cost of duplicating and storing ghost copies of data, and ii) allows for seamless sharing of analysis results amongst collaborators. We describe the framework and provide example analyses (e.g., sea-surface height variability, submesoscale vertical buoyancy fluxes, and comparison to predictions from the mixed-layer instability parametrization). Basin-to-global scale, submesoscale-permitting models are still at their early stage of development; their cost and carbon footprints are also rather large. It would, therefore, benefit the community to document the different model configurations for future best practices. We also argue that an emphasis on data analysis strategies would be crucial for improving the models themselves.

show abstract

“…Jupyter is also recognised by Hardisty and Wittenburg [9] as one possible CWFR solutions. Beg et al [29] also discuss the reproducibility of Jupyter notebooks as a scientific workflow. Jupyter notebook provides a one-study, one-document concept that is easily shareable.…”

Section: Notebook and Gitmentioning

confidence: 99%

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

et al. 2022

View full text Add to dashboard Cite

Machine learning (ML) applications in weather and climate are gaining momentum as big data and the immense increase in High-performance computing (HPC) power are paving the way. Ensuring FAIR data and reproducible ML practices are significant challenges for Earth system researchers. Even though the FAIR principle is well known to many scientists, research communities are slow to adopt them. Canonical Workflow Framework for Research (CWFR) provides a platform to ensure the FAIRness and reproducibility of these practices without overwhelming researchers. This conceptual paper envisions a holistic CWFR approach towards ML applications in weather and climate, focusing on HPC and big data. Specifically, we discuss Fair Digital Object (FDO) and Research Object (RO) in the DeepRain project to achieve granular reproducibility. DeepRain is a project that aims to improve precipitation forecast in Germany by using ML. Our concept envisages the raster datacube to provide data harmonization and fast and scalable data access. We suggest the Juypter notebook as a single reproducible experiment. In addition, we envision JuypterHub as a scalable and distributed central platform that connects all these elements and the HPC resources to the researchers via an easy-to-use graphical interface.

show abstract

Using Jupyter for Reproducible Scientific Workflows

Cited by 73 publications

References 11 publications

SWIRRL. Managing Provenance-aware and Reproducible Workspaces

SWIRRL. Managing Provenance-aware and Reproducible Workspaces

Cloud-based framework for inter-comparing submesoscale permitting realistic ocean models

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

Contact Info

Product

Resources

About