Managing Rapidly-Evolving Scientific Workflows

Freire, Juliana; Silva, Cláudio T.; Callahan, Steven P.; Santos, Emanuele; Scheidegger, Carlos; Vo, Huy T.

doi:10.1007/11890850_2

Cited by 188 publications

(149 citation statements)

References 8 publications

Supporting

Mentioning

145

Contrasting

Unclassified

Order By: Relevance

“…The provenance is generated by the middleware after inspection of the inputs and outputs of the wrapped components. This approach is popular with the scientific workflow community and includes systems such as Taverna [29], VisTrails [11], Kepler [2] and Wings [17]. Other middlewarebased systems like Karma [31] are not tied to a workflow system, but instead tap into the communication stack to capture provenance.…”

Section: Capturing Provenancementioning

confidence: 99%

Looking Inside the Black-Box: Capturing Data Provenance Using Dynamic Instrumentation

Stamatogiannakis¹,

Groth²,

Bos³

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Knowing the provenance of a data item helps in ascertaining its trustworthiness. Various approaches have been proposed to track or infer data provenance. However, these approaches either treat an executing program as a black-box, limiting the fidelity of the captured provenance, or require developers to modify the program to make it provenance-aware. In this paper, we introduce DataTracker, a new approach to capturing data provenance based on taint tracking, a technique widely used in the security and reverse engineering fields. Our system is able to identify data provenance relations through dynamic instrumentation of unmodified binaries, without requiring access to, or knowledge of, their source code. Hence, we can track provenance for a variety of well-known applications. Because DataTracker looks inside the executing program, it captures high-fidelity and accurate data provenance.

show abstract

Section: Capturing Provenancementioning

confidence: 99%

Looking Inside the Black-Box: Capturing Data Provenance Using Dynamic Instrumentation

Stamatogiannakis¹,

Groth²,

Bos³

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…VisTrails is an open-source provenance management and scientifi c workfl ow system designed to integrate the best of both scientifi c workfl ow and scientifi c visualization systems that combines a provenance-enabled workfl ow system with powerful visualization techniques (Freire et al 2006). As a visualization system, VisTrails makes advanced scientifi c visualization techniques available to users allowing them to explore and compare diff erent visual representations of their data.…”

Section: Existing Framework: Vistrailsmentioning

confidence: 99%

“…A distinguishing feature of VisTrails is a comprehensive provenance infrastructure that maintains detailed history information, referred to as provenance, about the steps followed and data derived in the course of an exploratory task (Freire et al 2006). VisTrails maintains this provenance for resulting data products (e.g.…”

Section: Existing Framework: Vistrailsmentioning

confidence: 99%

VisTrails SAHM: visualization and workflow management for species habitat modeling

et al. 2013

View full text Add to dashboard Cite

Th e Software for Assisted Habitat Modeling (SAHM) has been created to both expedite habitat modeling and help maintain a record of the various input data, pre-and post-processing steps and modeling options incorporated in the construction of a species distribution model through the established workfl ow management and visualization VisTrails software. Th is paper provides an overview of the VisTrails:SAHM software including a link to the open source code, a table detailing the current SAHM modules, and a simple example modeling an invasive weed species in Rocky Mountain National Park, USA.Understanding where species will thrive is a useful and important consideration for resource managers concerned with either promoting (for threatened or endangered species) or controlling (for invasive or unwanted species). Th e fi eld of species distribution or habitat niche modeling is contributing to this understanding. With an increasing availability of both ecological data (Graham et al. 2004) and software packages to fi t ecological niche models (Phillips et al. 2006, Franklin 2009, Th uiller et al. 2009, Guo and Liu 2010, Peterson et al. 2011 as well as new tools to evaluate model performance (Allouche et al. 2006, Phillips and Elith 2010, Warren et al. 2010), researchers and land managers now have an unprecedented opportunity to explore many parameters and iterations for any given habitat niche modeling exercise. Each niche modeling technique has multiple parameters and options that can be adjusted and choices for input and output data. For habitat models that consider climate change, there are future climate projections from diff erent climate modeling centers and multiple emissions scenarios to consider (IPCC 2007). Land managers might want to evaluate diff erent biological responses; such as diff erent/multiple species or, for a given species, diff erent life cycles (e.g. breeding vs nesting habitat). Furthermore, it may be of interest to modify the spatial extent and spatial resolution of both input/ predictor layers and output/model results. With these options and others not listed here, the potential number of model runs and related results can be overwhelming. Th ere is a need for careful documentation of the precise model confi guration as well as meaningful interpretation of results. Scientifi c workfl ow systems help address this need.

show abstract

“…Today, human activity is becoming increasingly automated with the aid of computational systems, which perform part or, in some cases, all of processes previously undertaken solely by humans. Interestingly, this means that activities in scientific [2], governmental [4] and commercial [12] settings are increasingly represented in computational systems, which presents an opportunity: it becomes possible to capture what happens in these settings both accurately and comprehensively.…”

Section: Motivationmentioning

confidence: 99%

“…what other actors state about the source, the actor's past performance, the content of the documentation. 2 The key to making this judgement is to know the creator of the statement. Therefore, it is fundamental that documentation about the past in computer systems be attributable, which is our second principle.…”

Section: Definition 2 (Evidence) "Evidence Is Information Whether Imentioning

confidence: 99%

Principles of High Quality Documentation for Provenance: A Philosophical Discussion

Groth

Miles

Munroe

2006

Provenance and Annotation of Data

View full text Add to dashboard Cite

Abstract.Computer technology enables the creation of detailed documentation about the processes that create or affect entities (data, objects, etc.). Such documentation of the past can be used to answer various kinds of questions regarding the processes that led to the creation or modification of a particular entity. The answer to such questions are known as an entity's provenance. In this paper, we derive a number of principles for documenting the past, grounded in work from philosophy and history, which allow for provenance questions to be answered within a computational context. These principles lead us to argue that an interaction-based model is particularly suited for representing high quality documentation of the past.History is important: in order to make progress in the future, it is important to learn the lessons of the past. History, therefore, becomes a vital resource for progressive societies. History is based on evidence or documentation of events that occurred in the past. Computer technology enables documentation to be produced that is more accurate and comprehensive than previously possible. Given the quantity of documentation that can be generated by computer systems (for example those running e-Science applications), what kind of documentation should be created that enables historians and users to most effectively answer questions they have about the past. To answer these questions, we argue for two principles for the creation of, what we term, high quality documentation of the past. One principle guarantees that users of documentation of the past have a precise semantics for it. The other principle guarantees that there exists a link between documentation and its creators. Together these principles are the first contribution of this paper.A question that is often posed by historians and users regards the provenance of an entity (object, data item, etc..): what was the process that led to the entity in question? To enable the answering of provenance questions in a manner that conforms with the aforementioned principles, we introduce a model of documentation based on the exchange of messages between actors. The justification of this model is the second contribution of this paper.The rest of this paper is organised as follows. We begin with a more detailed motivation of our work. After which, three assumptions about the world are

show abstract

Managing Rapidly-Evolving Scientific Workflows

Cited by 188 publications

References 8 publications

Looking Inside the Black-Box: Capturing Data Provenance Using Dynamic Instrumentation

Looking Inside the Black-Box: Capturing Data Provenance Using Dynamic Instrumentation

VisTrails SAHM: visualization and workflow management for species habitat modeling

Principles of High Quality Documentation for Provenance: A Philosophical Discussion

Contact Info

Product

Resources

About