A Real-Time Machine Learning and Visualization Framework for Scientific Workflows

Abstract

High-performance computing resources are currently widely used in science and engineering areas. Typical post-hoc approaches use persistent storage to save produced data from simulation, thus reading from storage to memory is required for data analysis tasks. For large-scale scientific simulations, such I/O operation will produce significant overhead. In-situ/in-transit approaches bypass I/O by accessing and processing in-memory simulation results directly, which suggests simulations and analysis applications should be more closely coupled. This paper constructs a flexible and extensible framework to connect scientific simulations with multi-steps machine learning processes and in-situ visualization tools, thus providing plugged-in analysis and visualization functionality over complex workflows at real time. A distributed simulation-time clustering method is proposed to detect anomalies from real turbulence flows.

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

“…In this approach, HTM model is used for each metric in the system to detect CPU or I/O anomalies in real-time execution. An online parallel ML algorithm for anomaly detection in computational fluid dynamics (CFD) and turbulence analysis introduced by [130]. This study integrates simulation applications with various analysis applications with the use of DataSpaces [29] as a flexible interaction and coordination in distributed and virtual shared space.…”

Building a scientific workflow framework to enable real‐time machine learning and visualization

“…In this paper, we extend our previous work and design a computational fluid dynamics (CFD) specific machine learning method to automatically detect anomaly flows. In addition to targeting at automated data analysis (ie, one of the objectives of this work), we also aim to expedite the process of online simulation‐time data analysis.…”