Workflow Management in Condor

Couvares, P.; Kosar, Tevfik; Roy, Alain; Weber, Jeff; Wenger, Kent

doi:10.1007/978-1-84628-757-2_22

Cited by 123 publications

(89 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…WorkflowSim is tightly bound to the SWfMS Pegasus [20] and adds to CloudSim (1) the workflow engine underlying Pegasus and DagMan [41], (2) an elaborate model of node failures, (3) a model of delays occurring in the various levels of the Pegasus stack (e.g., queue delays, pre/post-processing delays, data transfer delays), and (4) the implementations of several workflow schedulers implemented in Pegasus (e.g., greedy task queue, HEFT [23], Min-Min, and Max-Min [24]). Parameters are directly learned from traces of real executions.…”

Section: Related Workmentioning

confidence: 99%

DynamicCloudSim: Simulating heterogeneity in computational clouds

Bux

Leser

2015

Future Generation Computer Systems

View full text Add to dashboard Cite

Simulation has become a commonly employed first step in evaluating novel approaches towards resource allocation and task scheduling on distributed architectures. However, existing simulators fall short in their modeling of the instability common to shared computational infrastructure, such as public clouds. In this work, we present DynamicCloudSim which extends the popular simulation toolkit CloudSim with several factors of instability, including inhomogeneity and dynamic changes of performance at runtime as well as failures during task execution. As a use case and validation of the introduced functionality, we simulate the impact of instability on scientific workflow scheduling by assessing and comparing the performance of four schedulers in the course of several experiments. Results indicate that our model seems to adequately capture the most important aspects of cloud performance instability, though a validation on real hardware is still pending. The source code of DynamicCloudSim and the examined schedulers is available at https://code.google.com/p/dynamiccloudsim/.

show abstract

Section: Related Workmentioning

confidence: 99%

DynamicCloudSim: Simulating heterogeneity in computational clouds

Bux

Leser

2015

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…Initially, with GPUs in a testing phase, this was accomplished locally at a few sites with DAGMan for HTCondor [8]. This worked fairly reliably, but did not extend to other job submission systems and did not allow jobs from one site to utilize GPUs at another site.…”

Section: Iceprod V1mentioning

confidence: 99%

IceCube simulation production and the transition to IceProd2

Schultz

2016

EPJ Web of Conferences

View full text Add to dashboard Cite

Abstract. IceCube's simulation production relies largely on dynamic, heterogeneous resources spread around the world. Datasets consist of many thousands of job workflow subsets running in parallel as directed acyclic graphs (DAGs) and using varying resources. IceProd is a set of Python daemons which process job workflow and maintain configuration and status information on jobs before, during, and after processing. IceProd manages a complex workflow of DAGs to distribute jobs across all computing grids and optimize resource usage. IceProd2 is a new version of IceProd with substantial increases in security, reliability, scalability, and ease of use. It is undergoing testing and will be deployed this fall.

show abstract

“…HTTP requests are an example [31]: some resources (e.g., images) are requested only because they are referenced by others (web pages). Job-and task-level workloads are likely to have this flavor, too: users may use higher-level tools (such as [32]- [35]) to write workflows on top of a lower-level scheduler from which the trace is extracted. Unless the obfuscation is done with care, the workflow's dependency information is easy to cull by mistake.…”

Section: A Unavailable Informationmentioning

confidence: 99%

Obfuscatory obscanturism: Making workload traces of commercially-sensitive systems safe to release

Reiss

Wilkes

Hellerstein

2012

2012 IEEE Network Operations and Management Symposium

View full text Add to dashboard Cite

Abstract-Cloud providers such as Google are interested in fostering research on the daunting technical challenges they face in supporting planetary-scale distributed systems, but no academic organizations have similar scale systems on which to experiment. Fortunately, good research can still be done using traces of real-life production workloads, but there are risks in releasing such data, including inadvertently disclosing confidential or proprietary information, as happened with the Netflix Prize data. This paper discusses these risks, and our approach to them, which we call systematic obfuscation. It protects proprietary and personal data while leaving it possible to answer interesting research questions. We explain and motivate some of the risks and concerns and propose how they can best be mitigated, using as an example our recent publication of a monthlong trace of a production system workload on a 11k-machine cluster.

show abstract

Workflow Management in Condor

Cited by 123 publications

References 14 publications

DynamicCloudSim: Simulating heterogeneity in computational clouds

DynamicCloudSim: Simulating heterogeneity in computational clouds

IceCube simulation production and the transition to IceProd2

Obfuscatory obscanturism: Making workload traces of commercially-sensitive systems safe to release

Contact Info

Product

Resources

About