The MPO API: A tool for recording scientific workflows

Wright, J. C.; Greenwald, M.; Stillerman, J.; Abla, G.; Chanthavong, Bobby; Flanagan, S.; Schissel, D.P.; Lee, Xia; Romosan, Alex; Shoshani, Arie

doi:10.1016/j.fusengdes.2014.02.011

Cited by 4 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Others, such as those from the HEP and BES communities are not aware of any widely adopted community standards on provenance. While much research has focused on provenance capture and storage methods, workflow management system developers have been at the forefront of integrating automated provenance capture into their software [162]. Science communities on the other hand often have a long tradition of capturing the provenance of their work in non-digital means, such as Lab Notebooks mentioned a number of times by BES scientists at the workshop.…”

Section: Metadata Best Practicesmentioning

confidence: 99%

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Bethel¹,

eds²

2016

View full text Add to dashboard Cite

The Department of Energy (DOE) Office of Science (SC) operates dozens of national science user facilities that span many different disciplines. These facilities include accelerators, colliders, supercomputers, light sources and neutron sources, as well as facilities for studying the nanoworld, the environment, and the atmosphere. In Fiscal Year 2014 over 33,000 researchers from academia, industry, and government laboratories, spanning all fifty states and the District of Columbia, utilized these unique facilities to perform new scientific research. Each of these facilities generates vast amounts of scientific data, and the rate, size, and complexity of this data is rapidly increasing, thanks to advances in technology. A growing concern, which motivates this workshop, is the likely significant adverse impact on science programs that will result without significant advances in the capabilities needed to manage and gain knowledge from these collections of data. The purpose of this workshop, held 29 September 2015 through 1 October 2015 in Bethesda, MD, is to help the Advanced Scientific Computing Research (ASCR) and research community better understand needs related to the management, analysis, and visualization of experimental and observational data (EOD) collected and generated by experimental and observational science projects (EOS) at Office of Science user facilities. The science needs articulated in this report, along with the findings, recommendations, and detailed discussion of issues, collectively are consistent with and show opportunity for cultivating a research, development, and deployment path that takes steps towards realizing the vision articulated in the National Strategic Computing Initiative (NCSI) [1, 2] and the Big Data Research and Development Initiative [3, 4]. Specifically, the science use cases reveal a trend towards the convergence of data and computing: data-and compute-centric needs and opportunities are increasingly intertwined, interrelated, and symbiotic. Advances in our ability to collect data in turn require advances in computational capabilities to understand, preserve, share and make optimal use of data, and can even favorably impact the quality and value of science we perform by improving the quality of data we collect. This workshop report consists of input from a set of representatives from DOE EOS facilities and researchers in mathematics and computer science. The findings, drawn from use cases that articulate science drivers, indicate acute and urgent needs. This report articulates a path forward for meeting those needs, a path that includes advances in mathematics, computer and data science, as well as advances in and the use of HPC computational and networking infrastructures. One major theme recurring in the use cases is that individual EOS projects are presently pursuing their own independent paths towards meeting data-centric challenges, which results in the duplication of effort and increased costs across the entire program. EOS projects, and the EO community as a whole, ...

show abstract

Section: Metadata Best Practicesmentioning

confidence: 99%

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Bethel¹,

eds²

2016

View full text Add to dashboard Cite

show abstract

“…If the analysis is orchestrated using a workflow system, provenance information is automatically captured in a separate data store, which may be a relational database management system. Existing systems such as MPO (Metadata, Provenance, and Ontology) [52] is an example of this approach. The representation and storage of provenance information is specific to the workflow system used by the scientist.…”

Section: Acquire and Propagate Insightsmentioning

confidence: 99%

“…Should a scientist desire to access the provenance metadata, they first need to learn the data model and the query interface of the workflow system. Some workflow systems store provenance metadata in a relational database and scientists can query the database using SQL [52]. Other systems use representations that are optimized for provenance metadata, and scientists query provenance metadata through a system-specific API [25].…”

Section: A Classification Of Metadata Informationmentioning

confidence: 99%

See 1 more Smart Citation

Towards Exascale Scientific Metadata Management

Blanas,

Byna

2015

Preprint

View full text Add to dashboard Cite

Advances in technology and computing hardware are enabling scientists from all areas of science to produce massive amounts of data using large-scale simulations or observational facilities. In this era of data deluge, effective coordination between the data production and the analysis phases hinges on the availability of metadata that describe the scientific datasets. Existing workflow engines have been capturing a limited form of metadata to provide provenance information about the identity and lineage of the data. However, much of the data produced by simulations, experiments, and analyses still need to be annotated manually in an ad hoc manner by domain scientists. Systematic and transparent acquisition of rich metadata becomes a crucial prerequisite to sustain and accelerate the pace of scientific innovation. Yet, ubiquitous and domain-agnostic metadata management infrastructure that can meet the demands of extreme-scale science is notable by its absence.To address this gap in scientific data management research and practice, we present our vision for an integrated approach that (1) automatically captures and manipulates information-rich metadata while the data is being produced or analyzed and (2) stores metadata within each dataset to permeate metadataoblivious processes and to query metadata through established and standardized data access interfaces. We motivate the need for the proposed integrated approach using applications from plasma physics, climate modeling and neuroscience, and then discuss research challenges and possible solutions.

show abstract

“…The motivation of the project and design principles of the prototype system have been described earlier [1,2,3]. This paper discusses the most recent results of the MPO project's research, including the current status of the MPO software system and deployment experiences.…”

Section: Introductionmentioning

confidence: 99%

The MPO system for automatic workflow documentation

Abla

Coviello

Flanagan

et al. 2016

Fusion Engineering and Design

View full text Add to dashboard Cite

Data from large-scale experiments and extreme-scale computing is expensive to produce and may be used for critical applications. However, it is not the mere existence of data that is important, but our ability to make use of it. Experience has shown that when metadata is better organized and more complete, the underlying data becomes more useful. Traditionally, capturing the steps of scientific workflows and metadata was the role of the lab notebook, but the digital era has resulted instead in the fragmentation of data, processing, and annotation. This paper presents the Metadata, Provenance, and Ontology (MPO) System, the software that can automate the documentation of scientific workflows and associated information. Based on recorded metadata, it provides explicit information about the relationships among the elements of workflows in notebook form augmented with directed acyclic graphs. A set of web-based graphical navigation tools and Application Programming Interface (API) have been created for searching and browsing, as well as programmatically accessing the workflows and data. We describe the MPO concepts and its software architecture. We also report the current status of the software as well as the initial deployment experience.

show abstract

The MPO API: A tool for recording scientific workflows

Cited by 4 publications

References 3 publications

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Towards Exascale Scientific Metadata Management

The MPO system for automatic workflow documentation

Contact Info

Product

Resources

About