Building a scientific workflow framework to enable real‐time machine learning and visualization

Li, Feng; Song, Fengguang

doi:10.1002/cpe.4703

Cited by 8 publications

(5 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, in-situ processing has been used to accelerate the loop of simulation-analysis by eliminating the need of writing intermediate data back into files for analytic [95], [96]. Finally, ML techniques have been utilized to perform CFD simulation in a parallel and online mode [97]. For computationally expensive simulations, ML techniques can prioritize jobs, reduce the search space, predict failure in simulation [98] and help with prediction in favor of faster simulations.…”

Section: B Scientific Simulation and Analysismentioning

confidence: 99%

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Nouri¹,

Davis²,

Subedi³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this survey, we discuss the challenges of executing scientific workflows as well as existing Machine Learning (ML) techniques to alleviate those challenges. We provide the context and motivation for applying ML to each step of the execution of these workflows. Furthermore, we provide recommendations on how to extend ML techniques to unresolved challenges in the execution of scientific workflows. Moreover, we discuss the possibility of using ML techniques for in-situ operations. We explore the challenges of in-situ workflows and provide suggestions for improving the performance of their execution using ML techniques.

show abstract

Section: B Scientific Simulation and Analysismentioning

confidence: 99%

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Nouri¹,

Davis²,

Subedi³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent works in the literature have tackled the anomaly detection problem (Deelman et al, 2019; Rodriguez et al, 2018; Li and Song 2019; Singh et al, 2018; Papadimitriou et al, 2019). However, they have limited capability in correlating data from infrastructure sources, with each other, and with application-level data.…”

Section: Introductionmentioning

confidence: 99%

“…Several machine learning (ML) and deep learning (DL) approaches have been applied more recently to address the anomaly detection problem. Although Li and Song (2019) use DL to forecast anomalies in high-energy physics jobs by leveraging minimal data during early parts of the job’s execution and achieved improvements of up to 14% in resource utilization, this work only considers application-level metrics. Similarly, Wang et al (2020) also used only application-level metrics to apply clustering and decision trees to detect anomalies.…”

Section: Introductionmentioning

confidence: 99%

Graph neural networks for detecting anomalies in scientific workflows

Jin

Krishnan

Papadimitriou

et al. 2023

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Identifying and addressing anomalies in complex, distributed systems can be challenging for reliable execution of scientific workflows. We model these workflows as directed acyclic graphs (DAGs), where the nodes and edges of the DAGs represent jobs and their dependencies, respectively. We develop graph neural networks (GNNs) to learn patterns in the DAGs and to detect anomalies at the node (job) and graph (workflow) levels. We investigate workflow-specific GNN models that are trained on a particular workflow and workflow-agnostic GNN models that are trained across the workflows. Our GNN models, which incorporate both individual job features and topological information from the workflow, show improved accuracy and efficiency compared to conventional learning methods for detecting anomalies. While joint trained with multiple scientific workflows, our GNN models reached an accuracy more than 80% for workflow level and 75% for job level anomalies. In addition, we illustrate the importance of hyperparameter tuning method in our study that can significantly improve the metric(s) measure of evaluating the GNN models. Finally, we integrate explainable GNN methods to provide insights on job features in the workflow that cause an anomaly.

show abstract

“…During turbulent times, it is much harder to successfully manage a business (Naujoks, 2010). In difficult times, you can succeed with an excellent knowledge set (Anderson et al, 2018) and an excellent set of managerial tools (Li and Song, 2019). It has long been known that models can help with the sustainability of the business if they are prepared in the right way (Stafford et al, 1999).…”

Section: Introductionmentioning

confidence: 99%

Family Business - Sustainability Model

Svoboda¹

2020

AOP

View full text Add to dashboard Cite

Family firms form a considerable global economic pillar and have a long tradition. Due to their restricted ownership structure and extended value structure, they deal with specific managerial challenges. In comparison with their generally owned competitors, family firms tend to be more stable, accountable and trustable. On the other hand, they have to combine private and business life, deal with succession issues and the relative closeness of the top management while resolving nonstandard governance mechanisms. The main objective of this paper is to propose how to create a dynamic model that can further propose viable strategies for the long-term sustainability of family businesses that maximise the stakeholders` utility. To be able to understand the internal dynamics of family businesses, as well as to fully incorporate the changing external environment, an underlying systemic model is required. It is proposed to incorporate several gradually evolving and mutually interlinked stages and to formalise knowledge acquired from the present literature extended with available data. The initial modelling stage, the mind mapping, summarises the key terminology and outlines the basic structure. These findings are subsequently refined through system and causal loop diagrams. The latter allows the formulation, discussion and generalisation of behavioural hypotheses related to the researched sustainability phenomenon. It was found that standard techniques and strategic planning tools do not fully support all the abovementioned aspects of family businesses. Thus, these specific features, reflecting various aspects of "familiness" were modelled and incorporated into a general framework for strategic planning and management. The proposed paper deals with specific internal matters of family-owned firms. It suggests and justifies a transparent model for their implementation, thus eliminating the major bottlenecks. With this knowledge, the addressed companies can sustainably develop their businesses and keep the quality of their private life on a desirable level.

show abstract

Building a scientific workflow framework to enable real‐time machine learning and visualization

Cited by 8 publications

References 29 publications

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Graph neural networks for detecting anomalies in scientific workflows

Family Business - Sustainability Model

Contact Info

Product

Resources

About