The role of machine learning in scientific workflows

Deelman, Ewa; Mandal, Anirban; Jiang, Ming; Sakellariou, Rizos

doi:10.1177/1094342019852127

Cited by 27 publications

(18 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The workflow model [10] is used to define the workflow structure and is of two types; abstract and concrete [11]. Workflow composition system helps user to add different components to the workflow [2,12,13].…”

Section: Figurementioning

confidence: 99%

Multi-Dependency and Time Based Resource Scheduling Algorithm for Scientific Applications in Cloud Computing

2021

View full text Add to dashboard Cite

Workflow scheduling is one of the significant issues for scientific applications among virtual machine migration, database management, security, performance, fault tolerance, server consolidation, etc. In this paper, existing time-based scheduling algorithms, such as first come first serve (FCFS), min–min, max–min, and minimum completion time (MCT), along with dependency-based scheduling algorithm MaxChild have been considered. These time-based scheduling algorithms only compare the burst time of tasks. Based on the burst time, these schedulers, schedule the sub-tasks of the application on suitable virtual machines according to the scheduling criteria. During this process, not much attention was given to the proper utilization of the resources. A novel dependency and time-based scheduling algorithm is proposed that considers the parent to child (P2C) node dependencies, child to parent node dependencies, and the time of different tasks in the workflows. The proposed P2C algorithm emphasizes proper utilization of the resources and overcomes the limitations of these time-based schedulers. The scientific applications, such as CyberShake, Montage, Epigenomics, Inspiral, and SIPHT, are represented in terms of the workflow. The tasks can be represented as the nodes, and relationships between the tasks can be represented as the dependencies in the workflows. All the results have been validated by using the simulation-based environment created with the help of the WorkflowSim simulator for the cloud environment. It has been observed that the proposed approach outperforms the mentioned time and dependency-based scheduling algorithms in terms of the total execution time by efficiently utilizing the resources.

show abstract

Section: Figurementioning

confidence: 99%

Multi-Dependency and Time Based Resource Scheduling Algorithm for Scientific Applications in Cloud Computing

2021

View full text Add to dashboard Cite

show abstract

“…Economic factors (including storage costs) for workflow execution are discussed in [1]. In [7], the authors discuss the role of Machine Learning (ML) for workflow execution and elaborate a general potential for resource provisionings such as optimisation of runtime parameters, data movements, and hierarchical storage. In [25], an ML model that stages data for in-situ analysis by exploiting the access patterns is introduced.…”

Section: State-of-the-artmentioning

confidence: 99%

Potential of I/O Aware Workflows in Climate and Weather

Kunkel

Pedro

2020

JSFI

View full text Add to dashboard Cite

The efficient, convenient, and robust execution of data-driven workflows and enhanced data management are essential for productivity in scientific computing. In HPC, the concerns of storage and computing are traditionally separated and optimised independently from each other and the needs of the end-to-end user. However, in complex workflows, this is becoming problematic. These problems are particularly acute in climate and weather workflows, which as well as becoming increasingly complex and exploiting deep storage hierarchies, can involve multiple data centres. The key contributions of this paper are: 1) A sketch of a vision for an integrated data-driven approach, with a discussion of the associated challenges and implications, and 2) An architecture and roadmap consistent with this vision that would allow a seamless integration into current climate and weather workflows as it utilises versions of existing tools (ESDM, Cylc, XIOS, and DDN's IME). The vision proposed here is built on the belief that workflows composed of data, computing, and communication-intensive tasks should drive interfaces and hardware configurations to better support the programming models. When delivered, this work will increase the opportunity for smarter scheduling of computing by considering storage in heterogeneous storage systems. We illustrate the performance-impact on an example workload using a model built on measured performance data using ESDM at DKRZ.

show abstract

“…This kind of provenance consumption, which often occurs post mortem, i.e., after workflow execution, is characterized as offline provenance analysis. A characterization of provenance analysis to leverage ML in support of workflows is surveyed by Deelman et al [7]. We propose here a taxonomy to classify provenance analysis in support of ML, by considering three classes: data, execution timing, and training timing.…”

Section: Characterizing Provenance Analysis In ML For Csementioning

confidence: 99%

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

Souza¹,

Valduriez²,

Mattoso³

et al. 2019

2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)

View full text Add to dashboard Cite

Machine Learning (ML) has become essential in several industries. In Computational Science and Engineering (CSE), the complexity of the ML lifecycle comes from the large variety of data, scientists' expertise, tools, and workflows. If data are not tracked properly during the lifecycle, it becomes unfeasible to recreate a ML model from scratch or to explain to stakeholders how it was created. The main limitation of provenance tracking solutions is that they cannot cope with provenance capture and integration of domain and ML data processed in the multiple workflows in the lifecycle, while keeping the provenance capture overhead low. To handle this problem, in this paper we contribute with a detailed characterization of provenance data in the ML lifecycle in CSE; a new provenance data representation, called PROV-ML, built on top of W3C PROV and ML Schema; and extensions to a system that tracks provenance from multiple workflows to address the characteristics of ML and CSE, and to allow for provenance queries with a standard vocabulary. We show a practical use in a real case in the O&G industry, along with its evaluation using 48 GPUs in parallel.Index Terms-Machine Learning Lifecycle, Workflow Provenance, Computational Science and Engineering (ii) PROV-ML: a new data representation, which combines W3C PROV [18] with W3C ML Schema [19], for prove-R. Souza et al. Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering.

show abstract

The role of machine learning in scientific workflows

Cited by 27 publications

References 77 publications

Multi-Dependency and Time Based Resource Scheduling Algorithm for Scientific Applications in Cloud Computing

Multi-Dependency and Time Based Resource Scheduling Algorithm for Scientific Applications in Cloud Computing

Potential of I/O Aware Workflows in Climate and Weather

Provenance Data in the Machine Learning Lifecycle in Computational Science and Engineering

Contact Info

Product

Resources

About