Swift: Fast, Reliable, Loosely Coupled Parallel Computation

Zhao, Yong; Hategan, Mihael; Clifford, Ben; Foster, Ian; Laszewski, Gregor von; Nefedova, Veronika; Raicu, Ioan; Stef‐Praun, Tiberiu; Wilde, Michael

doi:10.1109/services.2007.63

Cited by 288 publications

(223 citation statements)

References 11 publications

Supporting

Mentioning

219

Contrasting

Unclassified

Order By: Relevance

“…Based on a comprehensive study of the workflow literature from an architectural perspective and our own experience from the development of the VIEW system [10][16] [17] and Swift [31], we identify the following seven key architectural requirements for an SWFMS: (R1) User interface customizability and user interaction support; (R2) Reproducibility support; (R3) Heterogeneous and distributed services and software tools integration; (R4) Heterogeneous and distributed data product management; (R5) High-end computing support; (R6) Workflow monitoring and failure handling; and (R7) Interoperability.…”

Section: A Architectural Challengementioning

confidence: 99%

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Zhao

Fei

Raicu

et al. 2011

2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

Abstract-Cloud computing is gaining tremendous momentum in both academia and industry. The application of Cloud computing, however, has mostly focused on Web applications and business applications; while the recognition of using Cloud computing to support large-scale workflows, especially dataintensive scientific workflows on the Cloud is still largely overlooked. We coin the term "Cloud Workflow", to refer to the specification, execution, provenance tracking of large-scale scientific workflows, as well as the management of data and computing resources to enable the execution of scientific workflows on the Cloud. In this paper, we analyze why there has been such a gap between the two technologies, and what it means to bring Cloud and workflow together; we then present the key challenges in running Cloud workflow, and discuss the research opportunities in realizing workflows on the Cloud.

show abstract

Section: A Architectural Challengementioning

confidence: 99%

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Zhao

Fei

Raicu

et al. 2011

2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

show abstract

“…A number of independent efforts have developed SWFMSs, however, here we discuss Kepler [7], Taverna [8], Triana [17], Vistrails [14], Pegaus [13] and Swift [18]. Moreover, our implementation work has been performed within the Kepler environment.…”

Section: Scientific Workflow Enginesmentioning

confidence: 99%

Integrating Scientific Workflows and Large Tiled Display Walls: Bridging the Visualization Divide

Nguyen

Abramson

Bethwaite

et al. 2011

2011 40th International Conference on Parallel Processing Workshops

View full text Add to dashboard Cite

Abstract-Modern in-silico science (or e-Science) is a complex process, often involving multiple steps conducted across different computing environments. Scientific workflow tools help scientists automate, manage and execute these steps, providing a robust and repeatable research environment. Increasingly workflows generate data sets that require scientific visualization, using a range of display devices such as local workstations, immersive 3D caves and large display walls. Traditionally, this display step handled outside the workflow, and output files are manually copied to a suitable visualization engine for display. This inhibits the scientific discovery process disconnecting the workflow that generated the data from the display and interpretation processes. In this paper we present a solution that links scientific workflows with a variety of display devises, including large tiled display walls. We demonstrate the feasibility of the system by a prototype implementation that leverages the Kepler workflow engine and the SAGE display software. We illustrate the use of the system with a case study in workflow driven microscopy.

show abstract

“…The Swift [123] and Chimera [45] systems introduce a Virtual Data System (VDS) consisting of a set of relations to store the description of executable programs as transformations, their actual invocations as derivations, and inputs/outputs as data objects. These systems use provenance for tracking the data derivation history, on-demand data generation and re-generation, and data product validation.…”

Section: Storing and Querying Scientific Workflow Provenancementioning

confidence: 99%

“…With recent advances in the development of Scientific Workflow Management Systems [34,39,46,69,70,80,123], scientists from various domains are able to automate their experiments using scientific workflows to achieve significant scientific discoveries via complex and distributed scientific computations. As a result, scientific workflow has emerged as a new field to address the new requirements from scientists [70,75].…”

Section: Introductionmentioning

confidence: 99%

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Chebotko¹,

Lu²,

Fei³

et al. 2010

Data & Knowledge Engineering

View full text Add to dashboard Cite

Provenance metadata has become increasingly important to support scientific discovery reproducibility, result interpretation, and problem diagnosis in scientific workflow environments. The provenance management problem concerns the efficiency and effectiveness of the modeling, recording, representation, integration, storage, and querying of provenance metadata. Our approach to provenance management seamlessly integrates the interoperability, extensibility, and inference advantages of Semantic Web technologies with the storage and querying power of an RDBMS to meet the emerging requirements of scientific workflow provenance management. In this paper, we elaborate on the design of a relational RDF store, called RDFPROV, which is optimized for scientific workflow provenance querying and management. Specifically, we propose: i) two schema mapping algorithms to map an OWL provenance ontology to a relational database schema that is optimized for common provenance queries; ii) three efficient data mapping algorithms to map provenance RDF metadata to relational data according to the generated relational database schema, and iii) a schema-independent SPARQL-to-SQL translation algorithm that is optimized on-the-fly by using the type information of an instance available from the input provenance ontology and the statistics of the sizes of the tables in the database. Experimental results are presented to show that our algorithms are efficient and scalable. The comparison with two popular relational RDF stores, Jena and Sesame, and two commercial native RDF stores, AllegroGraph and BigOWLIM, showed that our optimizations result in improved performance and scalability for provenance metadata management. Finally, our case study for provenance management in a real-life biological simulation workflow showed the production quality and capability of the RDFPROV system. Although presented in the context of scientific workflow provenance management, many of our proposed techniques apply to general RDF data management as well.

show abstract

Swift: Fast, Reliable, Loosely Coupled Parallel Computation

Cited by 288 publications

References 11 publications

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Integrating Scientific Workflows and Large Tiled Display Walls: Bridging the Visualization Divide

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Contact Info

Product

Resources

About