Designing the myExperiment Virtual Research Environment for the Social Sharing of Workflows

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

“…Some projects, such as myExperiment [32] and Wf4Ever [33], have devoted substantial efforts toward this approach.…”

Section: Scientific Workflowsmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

124

View full text Add to dashboard Cite

show abstract

“…MyExperiment [17] is a repository with which scientists can share their experiments with their community. These workflows can be of several different supported formats to solve diverse challenges from different scientific areas.…”

Section: Related Workmentioning

confidence: 99%

Maintaining a Science Gateway - Lessons Learned from MoSGrid

Zimmermann¹,

Grunzke²,

Krüger³

2017

Proceedings of the 50th Hawaii International Conference on System Sciences (2017)

View full text Add to dashboard Cite

show abstract

“…Collaborative views over the provenance of these executing scripts can be captured by explicit reuse of results from previous experiments. An interesting opportunity arises from the support for Scientific Research Objects [29] by myExperiment [10]. Applications such as portal environments can deploy the content of SROs in new ways and collaborative views over them.…”

Section: Related Workmentioning

confidence: 99%

“…Community portals [8], virtual laboratories [9], and Web2.0-based social networking and sharing environments [10] are popular platforms to establish a common infrastructure where community members can contribute to data, workflows and projects through their user spaces under generic governance rules. Workflows could be executed multiple times by one or more scientists, potentially from an end-user interface that combines several workflows.…”

Section: Introductionmentioning

confidence: 99%

Understanding Collaborative Studies through Interoperable Workflow Provenance

Altıntaş

Anand

Crawl

et al. 2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The provenance of a data product contains information about how the product was derived, and is crucial for enabling scientists to easily understand, reproduce, and verify scientific results. Currently, most provenance models are designed to capture the provenance related to a single run, and mostly executed by a single user. However, a scientific discovery is often the result of methodical execution of many scientific workflows with many datasets produced at different times by one or more users. Further, to promote and facilitate exchange of information between multiple workflow systems supporting provenance, the Open Provenance Model (OPM) has been proposed by the scientific workflow community. In this paper, we describe a new query model that captures implicit user collaborations. We show how this model maps to OPM and helps to answer collaborative queries, e.g., identifying combined workflows and contributions of users collaborating on a project based on the records of previous workflow executions. We also adopt and extend the high-level Query Language for Provenance (QLP) with additional constructs, and show how these extensions allow non-expert users to express collaborative provenance queries against this model easily and concisely. Furthermore, we adopt the Provenance Challenge 3 (PC3) workflows as a collaborative and interoperable usecase scenario, where different stages of the workflow are executed in three different workflow environmentsKepler, Taverna, and WSVLAM. Through this usecase, we demonstrate how we can establish and understand collaborative studies through interoperable workflow provenance.

show abstract

Designing the myExperiment Virtual Research Environment for the Social Sharing of Workflows

Abstract: Abstract

Cited by 151 publications

References 10 publications

A characterization of workflow management systems for extreme-scale applications

A characterization of workflow management systems for extreme-scale applications

Maintaining a Science Gateway - Lessons Learned from MoSGrid

Understanding Collaborative Studies through Interoperable Workflow Provenance

Contact Info

Product

Resources

About