Application skeletons: Construction and use in eScience

Merzky, André; Zhang, Zhao; Jha, Shantenu

doi:10.1016/j.future.2015.10.001

Cited by 12 publications

(8 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We call this type of abstract application a "Skeleton Application" [27], [28], and have built an open source tool (Application Skeleton [29]) that can create skeletons. Our tool is implemented as a parser that reads in a configuration file that specifies a skeleton application, and produces three groups of outputs: (1) Preparation Scripts: run to produce the input/output directories and input files for the skeleton application.…”

Section: A Application Abstractionsmentioning

confidence: 99%

Integrating Abstractions to Enhance the Execution of Distributed Applications

Turilli

Liu

Zhao

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

Self Cite

View full text Add to dashboard Cite

One of the factors that limits the scale, performance, and sophistication of distributed applications is the difficulty of concurrently executing them on multiple distributed computing resources. In part, this is due to a poor understanding of the general properties and performance of the coupling between applications and dynamic resources. This paper addresses this issue by integrating abstractions representing distributed applications, resources, and execution processes into a pilot-based middleware. The middleware provides a platform that can specify distributed applications, execute them on multiple resource and for different configurations, and is instrumented to support investigative analysis. We analyzed the execution of distributed applications using experiments that measure the benefits of using multiple resources, the late-binding of scheduling decisions, and the use of backfill scheduling.

show abstract

Section: A Application Abstractionsmentioning

confidence: 99%

Integrating Abstractions to Enhance the Execution of Distributed Applications

Turilli

Liu

Zhao

et al. 2016

2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [26], application skeletons are used to build synthetic workflows that represent real applications for benchmarking. In our previous work [10], we developed a tool for generating synthetic workflow configurations based on real-world workflow instances.…”

Section: Related Workmentioning

confidence: 99%

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

Coleman¹,

Casanova²,

Pottier³

et al. 2021

Preprint

View full text Add to dashboard Cite

Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in realworld workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions.

show abstract

“…These synthetic workflows are key for supporting the development and evaluation of workflow algorithms. Also, they can provide a fundamental building block for the automatic generation of workflow application skeletons [13], which can then be used to benchmark workflow systems.…”

Section: Related Workmentioning

confidence: 99%

WfChef: Automated Generation of Accurate Scientific Workflow Generators

Coleman¹,

Casanova²,

Silva³

2021

Preprint

View full text Add to dashboard Cite

Scientific workflow applications have become mainstream and their automated and efficient execution on largescale compute platforms is the object of extensive research and development. For these efforts to be successful, a solid experimental methodology is needed to evaluate workflow algorithms and systems. A foundation for this methodology is the availability of realistic workflow instances. Dozens of workflow instances for a few scientific applications are available in public repositories. While these are invaluable, they are limited: workflow instances are not available for all application scales of interest. To address this limitation, previous work has developed generators of synthetic, but representative, workflow instances of arbitrary scales. These generators are popular, but implementing them is a manual, labor-intensive process that requires expert application knowledge. As a result, these generators only target a handful of applications, even though hundreds of applications use workflows in production.In this work, we present WfChef, a framework that fully automates the process of constructing a synthetic workflow generator for any scientific application. Based on an input set of workflow instances, WfChef automatically produces a synthetic workflow generator. We define and evaluate several metrics for quantifying the realism of the generated workflows. Using these metrics, we compare the realism of the workflows generated by WfChef generators to that of the workflows generated by the previously available, hand-crafted generators. We find that the WfChef generators not only require zero development effort (because it is automatically produced), but also generate workflows that are more realistic than those generated by hand-crafted generators.

show abstract

Application skeletons: Construction and use in eScience

Cited by 12 publications

References 36 publications

Integrating Abstractions to Enhance the Execution of Distributed Applications

Integrating Abstractions to Enhance the Execution of Distributed Applications

WfCommons: A Framework for Enabling Scientific Workflow Research and Development

WfChef: Automated Generation of Accurate Scientific Workflow Generators

Contact Info

Product

Resources

About