Instrumentation and Trace Analysis for Ad-Hoc Python Workflows in Cloud Environments

Acuna, Ruben; Lacroix, Zoé; Bazzi, Rida A.

doi:10.1109/cloud.2015.25

Cited by 8 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Legacy workflows are first processed by the WISE method, first introduced in [4]. It captures the dataflow of an ad-hoc workflow (i.e., a provenance graph) through its execution and abstracts it with a process for simplifying repetitive regions.…”

Section: Methods Overviewmentioning

confidence: 99%

“…This representation provides a data-centric view of the ad-hoc workflow's structure, demonstrating its essential behavior, and which may be used for other purposes such as rewriting or transformation. Then, an equivalence analysis not presented in [4] and described in more details in Section IV in performed in order to identify and merge all similar events into a more abstract data dependency graph. The final step of the WISE method consists in simplifying the overall graph to produce a skeleton of the abstract data dependency graph.…”

Section: Methods Overviewmentioning

confidence: 99%

“…Second Iteration: The leading nodes are updated to include all nodes which are concrete and have only abstract import sys, os input_name = sys.argv [1] cmd = "split_multifasta.py -input " + input_name + " -outfolder dna" os.system(cmd) files = [f for f in os.listdir("./dna")] for fn in files: cmd = "dna2rna.py -inputfile dna/" + fn +" -outputfile rna/" + fn os.system(cmd) files = [f for f in os.listdir("./rna")] for fn in files: cmd = "rna2aa.py -inputfile rna/" + fn +" -outputfile aa/" + fn os.system(cmd) successors (4,5,6). The first of these nodes (4) is selected as a seed.…”

Section: Rule 2 (Flow Equivalent)mentioning

confidence: 99%

See 2 more Smart Citations

Extracting Semantics from Legacy Scientific Workflows

Acuna

Lacroix

2016

2016 IEEE Tenth International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Methods Overviewmentioning

confidence: 99%

Section: Methods Overviewmentioning

confidence: 99%

Section: Rule 2 (Flow Equivalent)mentioning

confidence: 99%

See 1 more Smart Citation

Extracting Semantics from Legacy Scientific Workflows

Acuna

Lacroix

2016

2016 IEEE Tenth International Conference on Semantic Computing (ICSC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several systems allow the collection of workflow-like provenance without requiring the user to use workflow tools. Acuña et al [9] describe a system that captures file-level provenance from ad hoc workflows written in Python. Burrito [10] collects provenance as a scientist works with multiple tools by inserting plug-ins that monitor shell commands, file activity and GUI activity.…”

Section: Related Workmentioning

confidence: 99%

Using Introspection to Collect Provenance in R

2018

View full text Add to dashboard Cite

Data provenance is the history of an item of data from the point of its creation to its present state. It can support science by improving understanding of and confidence in data.RDataTracker is an R package that collects data provenance from R scripts (https://github.com/End-to-end-provenance/RDataTracker). In addition to details on inputs, outputs, and the computing environment collected by most provenance tools, RDataTracker also records a detailed execution trace and intermediate data values. It does this using R's powerful introspection functions and by parsing R statements prior to sending them to the interpreter so it knows what provenance to collect. The provenance is stored in a specialized graph structure called a Data Derivation Graph, which makes it possible to determine exactly how an output value is computed or how an input value is used. In this paper, we provide details about the provenance RDataTracker collects and the mechanisms used to collect it. We also speculate about how this rich source of information could be used by other tools to help an R programmer gain a deeper understanding of the software used and to support reproducibility.

show abstract

“…Steps one and two were presented in more details in [37]. Tracing by WISE is similar to data provenance methods (e.g., [34,26]), while dataflow abstraction is similar to methods for reducing workflow complexity (e.g., [38,39]).…”

Section: Methodsmentioning

confidence: 99%

Managing and Documenting Legacy Scientific Workflows

Acuna

Chomilier

Lacroix

2015

Journal of Integrative Bioinformatics

View full text Add to dashboard Cite

SummaryScientific legacy workflows are often developed over many years, poorly documented and implemented with scripting languages. In the context of our cross-disciplinary projects we face the problem of maintaining such scientific workflows. This paper presents the Workflow Instrumentation for Structure Extraction (WISE) method used to process several ad-hoc legacy workflows written in Python and automatically produce their workflow structural skeleton. Unlike many existing methods, WISE does not assume input workflows to be preprocessed in a known workflow formalism. It is also able to identify and analyze calls to external tools. We present the method and report its results on several scientific workflows.

show abstract

Instrumentation and Trace Analysis for Ad-Hoc Python Workflows in Cloud Environments

Cited by 8 publications

References 29 publications

Extracting Semantics from Legacy Scientific Workflows

Extracting Semantics from Legacy Scientific Workflows

Using Introspection to Collect Provenance in R

Managing and Documenting Legacy Scientific Workflows

Contact Info

Product

Resources

About