Scheduling Workloads of Workflows with Unknown Task Runtimes

Ilyushkin, Alexey; Ghit, Bogdan; Epema, Dick

doi:10.1109/ccgrid.2015.27

Cited by 17 publications

(22 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using several time series analysis techniques such as Linear Regression, Auto Regressive Moving Average (ARMA), and Multiple Regression, ConPaaS predicts future demand, and provisions resources accordingly. 6) Token: is designed specifically to autoscale for workflows [21]. By using structural information from the DAG structure, and by propagating execution tokens in the workflow, this policy estimated the level of parallelism and derives the (lack of) need for resources.…”

Section: Provisioning Policiesmentioning

confidence: 99%

A Trace-Based Performance Study of Autoscaling Workloads of Workflows in Datacenters

Versluis

Neacsu

Iosup

2018

2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

To improve customer experience, datacenter operators offer support for simplifying application and resource management. For example, running workloads of workflows on behalf of customers is desirable, but requires increasingly more sophisticated autoscaling policies, that is, policies that dynamically provision resources for the customer. Although selecting and tuning autoscaling policies is a challenging task for datacenter operators, so far relatively few studies investigate the performance of autoscaling for workloads of workflows. Complementing previous knowledge, in this work we propose the first comprehensive performance study in the field. Using trace-based simulation, we compare state-of-the-art autoscaling policies across multiple application domains, workload arrival patterns (e.g., burstiness), and system utilization levels. We further investigate the interplay between autoscaling and regular allocation policies, and the complexity cost of autoscaling. Our quantitative study focuses not only on traditional performance metrics and on state-of-the-art elasticity metrics, but also on timeand memory-related autoscaling-complexity metrics. Our main results give strong and quantitative evidence about previously unreported operational behavior, for example, that autoscaling policies perform differently across application domains and by how much they differ.

show abstract

Section: Provisioning Policiesmentioning

confidence: 99%

A Trace-Based Performance Study of Autoscaling Workloads of Workflows in Datacenters

Versluis

Neacsu

Iosup

2018

2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

show abstract

“…We used three WF structures, namely Montage, LIGO and SIPHT [4], and four workloads. Three workloads consist only of a single WF type, and the fourth one contains an equal mixture of all the considered WF structures.…”

Section: Unknown Task Runtimesmentioning

confidence: 99%

“…First, we have proposed a set of online scheduling policies to schedule workloads of WFs in a distributed environment with unknown task runtimes [4]. We have investigated these policies with simulations with realistic synthetic workloads.…”

mentioning

confidence: 99%

“…There they allow to perform massively parallelized sequencing of millions of nucleic acids, and the amount of data generated by these WFs is often gigabytes to terabyte [3]. Among all the job types in the workloads of modern large computing environments such as clusters and datacenters, WFs constitute one of the most difficult types of jobs to schedule [4]. Their sizes can reach thousands or hundreds of thousands of tasks and are virtually unbounded, and the amount of required resources for a single WF can significantly fluctuate during its execution.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Towards a Realistic Scheduler for Mixed Workloads with Workflows

Ilyushkin

Epema

2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Many fields of modern science require huge amounts of computation, and workflows are a very popular tool in e-Science since they allow to organize many small, simple tasks to solve big problems. They are used in astronomy, bioinformatics, machine learning, social network analysis, physics, and many other branches of science. Workflows are notoriously difficult to schedule, and the vast majority of research on workflow scheduling is concerned with scheduling single workflows with known runtimes. The goal of this PhD research is to bring more realism to the problem of workflow scheduling in actual systems. First, in real systems, multiple workflows may be contending for the available resources. Second, task runtime estimates are not always known, and task runtime estimates may be wrong. Third, workflows are usually not the only type of jobs submitted to a system, there may for instance also be parallel applications and bags-oftasks. Accordingly, the purpose of this PhD research is to create and analyze policies for online scheduling of workloads of workflows with and without known task runtimes that also contain jobs of other types. We are in the process of simulating policies, and we will validate our results by means of an implementation and real-world experiments with the KOALA-W workflow processing system.

show abstract

“…The presented mechanism also takes budget and deadline constraints into account. Ilyushkin et al [23] create and analyze policies for scheduling of workflows with and without know task running-time. Relative to our work, the authors focus on a different SLO related with the task running-time instead of deadline constraints.…”

Section: General Workflow-schedulingmentioning

confidence: 99%