SWIRRL. Managing Provenance-aware and Reproducible Workspaces

Abstract

Modern interactive tools for data analysis and visualisation are designed to expose their functionalities as a service through the Web. We present in this paper a Web API (SWIRRL) that allows Virtual Research Environments (VREs) to easily integrate such tools in their websites and re-purpose them to their users. The API deals, on behalf of the clients, with the underlying complexity of allocating and managing resources within a target cloud platform. By combining storage and containerised services, offering analysis notebooks and other visualisation software, the API creates dedicated working sessions on-demand, which can be accessed collaboratively. Thanks to the API’s support for workflow execution, SWIRRL workspaces can be automatically populated with data of interest collected from external data providers. The system keeps track of updates and changes affecting the data and the tools by adopting versioning and standard provenance technologies. Users are provided with interactive controls enabling traceability and recovery actions, including the possibility of creating executable snapshots of their environments. SWIRRL is built in cooperation with two research infrastructures in the field of solid earth science and climate data modeling. We report on the particular adoptions and use cases.

Data Management in Distributed, Federated Research Infrastructures: The Case of EPOS

“…As for the integrated data provision, this poses interesting challenges when it comes to the development of the so-called Integrated Core Services Distributed: these are intended as distributed resources like computational resources (HPC), remote environment for data analysis and processing (e.g., Jupyter Notebooks), visualization tools (e.g., enlighten-web) and reproducible analysis environments (Spinuso et al 2022). This is an unexplored area, where first steps have been done on the technical side, but the sustainability and the governance dimension still need to be explored and established.…”

Research data infrastructures in environmental sciences—Challenges and implementation approaches focusing on the integration of software components

“…Moreover, researchers lack support in generating and updating provenance information directly from the analysis workflows, which hampers efforts to produce detailed provenance information. Some attempts have been made, for instance, by developing APIs (Spinuso et al., 2022). However, data users then require user‐friendly interactive visualization tools that facilitate the understanding and evaluation of complex provenance graphs.…”

The EPOS Research Infrastructure: a federated approach to integrate solid Earth science data and services

“…Operating the ICS-C implies succeeding in making the different data, data products and services provided by TCS interoperable and accessible to users for fostering multidisciplinary research in solid Earth science [Bailo et al 2022, this volume]. The ICS-D has been designed during the Implementation Phase (2015-2019) and a prototype is currently under implementation [Spinuso et al, 2021]…”