The MPO system for automatic workflow documentation

Abstract

Data from large-scale experiments and extreme-scale computing is expensive to produce and may be used for critical applications. However, it is not the mere existence of data that is important, but our ability to make use of it. Experience has shown that when metadata is better organized and more complete, the underlying data becomes more useful. Traditionally, capturing the steps of scientific workflows and metadata was the role of the lab notebook, but the digital era has resulted instead in the fragmentation of data, processing, and annotation. This paper presents the Metadata, Provenance, and Ontology (MPO) System, the software that can automate the documentation of scientific workflows and associated information. Based on recorded metadata, it provides explicit information about the relationships among the elements of workflows in notebook form augmented with directed acyclic graphs. A set of web-based graphical navigation tools and Application Programming Interface (API) have been created for searching and browsing, as well as programmatically accessing the workflows and data. We describe the MPO concepts and its software architecture. We also report the current status of the software as well as the initial deployment experience.

Metadata Framework for Assisting Experimental Planning and Evaluation at Wendelstein 7-X

“…Metadata are referred to as additional information describing the process and environment of the actual experiment data acquisition, such as the intention of an experiment program, the technical and organizational environment, or quality and supplementary experiment relevant information. With this definition, metadata are as relevant as the experiment data and represent an instrument for data interpretation, organization, and scientific queries as increasingly discussed for quite general use cases and very different data types [4], [5], [6] even for meta-level topics as data provenance itself [7], [8].…”

Remote third shift EAST operation: a new paradigm

“…Enhanced metadata to support the distributed scientific teams is necessary with an increased utilization of the collaboration zone and the synchronization of the primary and remote MDSplus repositories. Enhanced metadata is enabled by adding the MPO System, which models data tracking to automate documentation of scientific workflows and their associated information, to the collaboration zone's infrastructure [19]. The MPO system allows the results appearing in the EAST's MDSplus data system to be properly identified by capturing information from the creation and recording/importing of physical data, through various levels of analysis, data preparation, supercomputer code execution, storage, post processing, data exporting, and publication.…”