SCEC CyberShake Workflows—Automating Probabilistic Seismic Hazard Analysis Calculations

Maechling, P. J.; Deelman, Ewa; Zhao, Li; Mehta, Gaurang; Gupta, Nitin; Mehringer, John; Kesselman, Carl; Callaghan, S.; Okaya, D. A.; Francoeur, H.; Gupta, Vinay K.; Cui, Yifeng; Vahi, Karan; Jordan, Thomas H.; Field, Edward H.

doi:10.1007/978-1-84628-757-2_10

Cited by 63 publications

(57 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In order to obtain the workflows that were run, the Pegasus mapper was used to map high-level workflow descriptions onto distributed resources. Pegasus stands for Planning for Execution in Grids and is used for numerous applications in many areas of science [23][24][25], including Montage, an astronomy application which creates science-grade image mosaics of the sky. The workflow manager depends on Condor DAGMan, which launches workflow tasks and maintains the dependencies between them.…”

Section: Toolsmentioning

confidence: 99%

On the Use of Cloud Computing for Scientific Workflows

Hoffa

Mehta

Freeman

et al. 2008

2008 IEEE Fourth International Conference on eScience

Self Cite

316

185

View full text Add to dashboard Cite

show abstract

Section: Toolsmentioning

confidence: 99%

On the Use of Cloud Computing for Scientific Workflows

Hoffa

Mehta

Freeman

et al. 2008

2008 IEEE Fourth International Conference on eScience

Self Cite

316

185

View full text Add to dashboard Cite

show abstract

“…The priority system is exponential, meaning that it is better to complete a workflow with priority p than any number of workflows with lower priorities. This assumption reflects the reality of many scientific applications, for example CyberShake [26,37], where it is better to compute a seismic hazard at the location of a single tall building than for an entire unpopulated desert area.…”

Section: Execution Modelmentioning

confidence: 99%

“…We have selected workflows representing several different classes of applications [32]. The selected applications include: CyberShake [37], a data-intensive application used by the Southern California Earthquake Center to calculate seismic hazards, Montage [30], an I/O-bound workflow used by astronomers to generate mosaics of the sky, and Epigenomics and SIPHT (sRNA Identification Protocol using High-throughput Technology) [36], two CPU-bound bioinformatics workflows.…”

Section: Evaluation Proceduresmentioning

confidence: 99%

Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds

Bryk¹,

Malawski

Juve³

et al. 2015

J Grid Computing

Self Cite

View full text Add to dashboard Cite

This paper focuses on data-intensive workflows and addresses the problem of scheduling workflow ensembles under cost and deadline constraints in Infrastructure as a Service (IaaS) clouds. Previous research in this area ignores file transfers between workflow tasks, which, as we show, often have a large impact on workflow ensemble execution. In this paper we propose and implement a simulation model for handling file transfers between tasks, featuring the ability to dynamically calculate bandwidth and supporting a configurable number of replicas, thus allowing us to simulate various levels of congestion. The resulting model is capable of representing a wide range of storage systems available on clouds: from inmemory caches (such as memcached), to distributed file systems (such as NFS servers) and cloud storage (such as Amazon S3 or Google Cloud Storage). We observe that file transfers may have a significant impact on ensemble execution; for some applications up to 90 % of the execution time is spent on file transfers. Next, we propose and evaluate a novel scheduling algorithm that minimizes the number of transfers by taking advantage of data caching and file locality. We find that for data-intensive applications it performs better than other scheduling algorithms. Additionally, we modify the original scheduling algorithms to effectively operate in environments where file transfers take non-zero time.

show abstract

“…CyberShake [7] is a scientific workflow on grid computing which is used for Probabilistic Seismic Hazard Analysis (PSHA). CyberShake has been executed on grid-based computing environment at the Southern California Earthquake Center (SCEC) .…”

Section: Related Workmentioning

confidence: 99%