Execution time estimation for workflow scheduling

Chirkin, Artem; Belloum, Adam; Kovalchuk, Sergey V.; Makkes, Marc X.; Melnik, Mikhail; Visheratin, Alexander A.; Nasonov, Denis

doi:10.1016/j.future.2017.01.011

Cited by 73 publications

(41 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Typically, mean values yield high values of standard deviation (due to variations inherent to the application itself, or the system including external load), thus estimations may not be accurate. Task characteristics estimation is beyond the scope of this work, and sophisticated methods to provide accurate estimates can be found in [22,23,24,25]. However, this work intends to demonstrate that even using inaccurate estimation methods, our proposed process can cope with the poor estimates and still yield good results.…”

Section: Decision Agentmentioning

confidence: 99%

“…Although several works address task requirement estimations based on provenance data [22,23,24,25], accurate estimations are still challenging, and may be specific to a certain type of application. In [26], a prediction algorithm based on machine learning (Naïve Bayes classifier) is proposed to identify faults before they occur, and to apply preventive actions to mitigate the faults.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Silva

Filgueira

Deelman

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. Although the scientific community has addressed this challenge from both theoretical and practical approaches, failure prediction, detection, and recovery still raise many research questions. In this paper, we propose an approach inspired by the control theory developed as part of autonomic computing to predict failures before they happen, and mitigated them when possible. The proposed approach is inspired on the proportionalintegral-derivative controller (PID controller) control loop mechanism, which is widely used in industrial control systems, where the controller will react to adjust its output to mitigate faults. PID controllers aim to detect the possibility of a non-steady state far enough in advance so that an action can be performed to prevent it from happening. To demonstrate the feasibility of the approach, we tackle two common execution faults of large scale data-intensive workflows-data storage overload and memory overflow. We developed a simulator, which implements and evaluates simple standalone PID-inspired controllers to autonomously manage data and memory usage of a data-intensive bioinformatics workflow that consumes/produces over 4.4TB of data, and requires over 24TB of memory to run all tasks concurrently. Experimental results obtained via simulation indicate that workflow executions may significantly benefit from the controller-inspired approach, in particular under online and unknown conditions. Simulation results show that nearly-optimal executions (slowdown of 1.01) can be attained when using our proposed method, and faults are detected and mitigated far in advance of their occurrence.

show abstract

Section: Decision Agentmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Silva

Filgueira

Deelman

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…This section is sub-divided into three complementary areas related to optimising workflows performance: parallel computing [24], scheduling and planning [25,26,27,28], and data management [29,30].…”

Section: Performance and Optimisationmentioning

confidence: 99%

Scientific workflows: Past, present and future

Atkinson

Gesing

Montagnat

et al. 2017

Future Generation Computer Systems

101

View full text Add to dashboard Cite

International audienceThis special issue and our editorial celebrate 10 years of progress with data-intensive or scientific workflows. There have been very substantial advances in the representation of workflows and in the engineering of workflow management systems (WMS). The creation and refinement stages are now well supported, with a significant improvement in usability. Improved abstraction supports cross-fertilisation between different workflow communities and consistent interpretation as WMS evolve. Through such re-engineering the WMS deliver much improved performance, significantly increased scale and sophisticated reliability mechanisms. Further improvement is anticipated from substantial advances in optimisation. We invited papers from those who have delivered these advances and selected 14 to represent today's achievements and representative plans for future progress. This editorial introduces those contributions with an overview and categorisation of the papers. Furthermore, it elucidates responses from a survey of major workflow systems, which provides evidence of substantial progress and a structured index of related papers. We conclude with suggestions on areas where further research and development is needed and offer a vision of future research directions

show abstract

“…However, accurately predicting/estimating the program execution time is difficult to achieve in shared environments where system resources can dynamically change over time. Inaccurate predictions may lead to performance degradation [20]. Table 8 lists the characteristics (such as optimization algorithm, objective, and features) of scientific publications that use machine learning and/or meta-heuristics for static scheduling.…”

Section: Static Schedulingmentioning

confidence: 99%

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

et al. 2018

View full text Add to dashboard Cite

While modern parallel computing systems offer high performance, utilizing these powerful computing resources to the highest possible extent demands advanced knowledge of various hardware architectures and parallel programming models. Furthermore, optimized software execution on parallel computing systems demands consideration of many parameters at compile-time and run-time. Determining the optimal set of parameters in a given execution context is a complex task, and therefore to address this issue researchers have proposed different approaches that use heuristic search or machine learning. In this paper, we undertake a systematic literature review to aggregate, analyze and classify the existing software optimization methods for parallel computing systems. We review approaches that use machine learning or meta-heuristics for software optimization at compile-time and run-time. we discuss challenges and future research directions. The results of this study may help to better understand the state-of-the-art techniques that use machine learning and meta-heuristics to deal with the complexity of software optimization for parallel computing systems. Furthermore, it may aid in understanding the limitations of existing approaches and identification of areas for improvement.

show abstract

Execution time estimation for workflow scheduling

Cited by 73 publications

References 16 publications

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Scientific workflows: Past, present and future

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

Contact Info

Product

Resources

About