Parallel Scripting for Applications at the Petascale and Beyond

Wilde, Michael; Foster, Ian; Iskra, Kamil; Beckman, Pete; Zhang, Zhao; Espinosa, Allan; Hategan, Mihael; Clifford, Ben; Raicu, Ioan

doi:10.1109/mc.2009.365

Cited by 83 publications

(49 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data volumes are also increasing dramatically in physics, earth science, medicine, and many other disciplines. As for application complexity, a protein simulation problem [27] involves running many instances of a structure prediction simulation, each with different random initial conditions. The simulation uses an "iterative fixing" algorithm that performs multiple rounds, each involving many parallel Monte Carlo simulated annealing models of molecular moves with energy minimization.…”

Section: Opportunitiesmentioning

confidence: 99%

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Zhao

Fei

Raicu

et al. 2011

2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

Abstract-Cloud computing is gaining tremendous momentum in both academia and industry. The application of Cloud computing, however, has mostly focused on Web applications and business applications; while the recognition of using Cloud computing to support large-scale workflows, especially dataintensive scientific workflows on the Cloud is still largely overlooked. We coin the term "Cloud Workflow", to refer to the specification, execution, provenance tracking of large-scale scientific workflows, as well as the management of data and computing resources to enable the execution of scientific workflows on the Cloud. In this paper, we analyze why there has been such a gap between the two technologies, and what it means to bring Cloud and workflow together; we then present the key challenges in running Cloud workflow, and discuss the research opportunities in realizing workflows on the Cloud.

show abstract

Section: Opportunitiesmentioning

confidence: 99%

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Zhao

Fei

Raicu

et al. 2011

2011 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery

Self Cite

View full text Add to dashboard Cite

show abstract

“…For this purpose, we proposed Kairos [5], a security architecture for provenance that uses cryptographic timestamps [6] and digital signatures. We are working with the Swift [13] parallel scripting system to extend its provenance system [4] with appropriate security controls. As future work, we will investigate fine-grained access control techniques, and a data model to store and query security properties.…”

Section: Discussionmentioning

confidence: 99%

Towards a Threat Model for Provenance in e-Science

Gadelha

Mattoso

Wilde

et al. 2010

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Scientists increasingly rely on workflow management systems to perform large-scale computational scientific experiments. These systems often collect provenance information that is useful in the analysis and reproduction of such experiments. On the other hand, this provenance data may be exposed to security threats which can result, for instance, in compromising the analysis of these experiments, or in illegitimate claims of attribution. In this work, we describe our ongoing work to trace security requirements for provenance systems in the context of e-Science, and propose some security controls to fulfill them.

show abstract

“…We considered two -Kepler [9,4] and Swift [15,12] as possible frameworks to implement a parallel version of the AMWG diagnostic workflow. These tools take dramatically different approaches to defining and structuring execution.…”

Section: Background and Related Workmentioning

confidence: 99%

Parallel high-resolution climate data analysis using swift

Woitaszek

Dennis

Sines

2011

Proceedings of the 2011 ACM International Workshop on Many Task Computing on Grids and Supercomputers

View full text Add to dashboard Cite

Advances in software parallelism and high-performance systems have resulted in an order of magnitude increase in the volume of output data produced by the Community Earth System Model (CESM). As the volume of data produced by CESM increases, the single-threaded script-based software packages traditionally used to post-process model output data have become a bottleneck in the analysis process. This paper presents a parallel version of the CESM atmosphere model data analysis workflow implemented using the Swift scripting language.Using the Swift implementation of the workflow, the time to analyze a 10-year atmosphere simulation on a typical cluster is reduced from 95 to 32 minutes on a single 8-core node and to 20 minutes on two nodes. The parallelized workflow is then used to evaluate several new data-intensive computational systems that feature RAM-based and flashbased storage. Even when constraining parallelism to limit the amount of file system space used by intermediate temporary data, our results show that the Swift-based implementation significantly reduces data analysis time.

show abstract

Parallel Scripting for Applications at the Petascale and Beyond

Cited by 83 publications

References 12 publications

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Opportunities and Challenges in Running Scientific Workflows on the Cloud

Towards a Threat Model for Provenance in e-Science

Parallel high-resolution climate data analysis using swift

Contact Info

Product

Resources

About