A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Churchill, R.M.; Chang, Choong-Seock; Choi, Jong Youl; Wang, R.; Klasky, Scott; Kube, Ralph; Park, H.; Choi, Minjun; Park, J. S.; Wolf, Matthew; Hager, Robert; Ku, Seung-Hoe; Kampel, S.H.; Carroll, Thomas J.; Silber, K.; Dart, Eli; Cho, B. S.

doi:10.1080/15361055.2020.1851073

Cited by 6 publications

(7 citation statements)

References 32 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Large-scale data analysis for experimental diagnostics can be accelerated using data science and networking techniques to stream the data from the experiment to large, remote HPC centers. By working with data streams, and leveraging the large HPC compute resources, better and more data analysis can be performed, which can better inform fusion scientists between plasma shots on the best way to optimize the next shot [413]. A demonstration of this used the streaming framework DELTA [414] to stream ECEI diagnostic data from the KSTAR tokamak in Korea to the NERSC HPC center in the USA, and complete spectral analysis of all channel pairs using multiple CPUs on the Cori supercomputer.…”

Section: Diagnostics and Fusion Data Streamsmentioning

confidence: 99%

2022 Review of Data-Driven Plasma Science

Archibald¹,

Asif²,

Becker³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Data science and technology offer transformative tools and methods to science. This review article highlights latest development and progress in the interdisciplinary field of data-driven plasma science (DDPS). A large amount of data and machine learning algorithms go hand in hand. Most plasma data, whether experimental, observational or computational, are generated or collected by machines today. It is now becoming impractical for humans to analyze all the data manually. Therefore, it is imperative to train machines to analyze and interpret (eventually) such data as intelligently as humans but far more efficiently in quantity. Despite the recent impressive progress in applications of data science to plasma science and technology, the emerging field of DDPS is still in its infancy. Fueled by some of the most challenging problems such as fusion energy, plasmaprocessing of materials, and fundamental understanding of the universe through observable plasma phenomena, it is expected that DDPS continues to benefit significantly from the interdisciplinary marriage between plasma science and data science into the foreseeable future. CONTENTSI. Introduction 6 II. Fundamental Data Science 6 A. Introduction 6 B. Data Reduction/Compression 7 C. Dimensional reduction and sparse modeling 8 D. ML-enhanced modeling and simulation 9 E. ML Hardware and integration with models 10 F. Workflow Automation 11 G. Uncertainty Quantification 12 H. Visualization and Data Understanding 13 I. ML Control Theory 15 III. Basic Plasma Physics and Laboratory Experiments A. Introduction B. Spectroscopy, imaging and tomography C. Sparse measurement and noise D. Synthetic instruments and data E. Experimental data visualization F. High-rep rate laser experiments G. Charged particle beams H. Control and Optimisation of Plasma Accelerator Experiments I. Dusty and complex plasmas J. Physics and machine learning K. Challenges and outlook 5 IV. Magnetic Confinement Fusion 36 A. Introduction 36 B. Data-Driven Physics Models 36 C. Optimizing experimental workflows with data-driven methods 37 D. Diagnostics and Fusion Data Streams 38 E. Prediction of Tokamak Disruption 39 F. Surrogate models of fusion plasma 40 G. Magnetic Fusion Energy Data Challenges and Solutions 41 H. Data Science for extreme scale simulation 43 I. Challenges and outlook 44 V. Inertial confinement fusion and high-energy-density physics 45 A. Introduction 45 B. Representation learning for multimodal data 45 C. Transfer learning for simulation and experimen 47 D. Uncertainty quantification and Bayesian inference 47 E. High-performance computing and simulation acceleration 49 F. Design exploration and optimization 49 G. Self-driving experimental facilities 51 H. Challenges and outlook 52 VI. Space and astronomical plasmas 52 A. Introduction 52 B. Space and ground instruments 53 C. Space weather prediction 55 D. Transfer learning to improve historic data 56 E. Surrogate models of fluid closures using machine learning 56 F. Magnetic reconnection 58 G. Challenges and outlook 60 VII. Plasma tec...

show abstract

Section: Diagnostics and Fusion Data Streamsmentioning

confidence: 99%

2022 Review of Data-Driven Plasma Science

Archibald¹,

Asif²,

Becker³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The work discussed in this contribution focuses on measurements of KSTAR plasmas on Cori, located at the National Energy Research Scientific Compute Center. These sites are connected through the Energy Sciences Network and data transfer rate benchmarks are reported in Churchill et al [2021]. Second, Delta uses ADIOS2 Godoy et al [2020] as a streaming library.…”

Section: The Adaptable Near Real-time Analysis Frameworkmentioning

confidence: 99%

“…In this paper we are presenting a streaming data analysis framework that aims to connect fusion experiments with remote HPC resources Choi et al [2016], Ralph , , Churchill et al [2021]. The streaming paradigm implemented by this framework allows the big-and fast data generated by fusion experiments to be seamlessly analyzed using supercomputers as they are generated.…”

Section: Introductionmentioning

confidence: 99%

Near real-time streaming analysis of big fusion data

Kube,

Churchill,

Chang

et al. 2021

Preprint

View full text Add to dashboard Cite

While experiments on fusion plasmas produce high-dimensional data time series with ever increasing magnitude and velocity, data analysis has been lagging behind this development. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a highperformance compute center, Delta allows to perform demanding data analysis tasks in between plasma pulses. This article describe the modular and expandable software architecture of Delta and presents performance benchmarks of its individual components as well as of entire workflows. Our focus is on the streaming analysis of ECEi data measured at KSTAR on NERSCs supercomputers and we routinely achieve data transfer rates of about 500 MByte per second. We show that a demanding turbulence analysis workload can be distributed among multiple GPUs and executes in under 5 minutes. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based realtime data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. By providing session leaders and physics operators results of higher order data analysis using live visualization they may monitor the evolution of a long-pulse discharge in near real-time and may make more informed decision on how to configure the machine for the next shot.

show abstract

“…Increasing data volume and data rates are met by a dichotomy between growth in computing power and growth in capacity of the storage systems at high-performance computing (HPC) systems. Over the last 20 years, the floating point operations of all systems listed in the top 500 list have increased approximately tenfold every four years 6 . The performance of the I/O systems of the top 500 supercomputer has only been tracked for the last four years, and the peak bandwidth of the fastest system increased approximately four-fold 7 .…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we present a streaming data analysis framework that aims to connect fusion experiments with remote HPC resources [2,6,19,32]. The streaming paradigm implemented by this framework allows the data generated by fusion experiments to be seamlessly analyzed using supercomputers as they are generated.…”

Section: Introductionmentioning

confidence: 99%

Near real-time streaming analysis of big fusion data

Kube¹,

Churchill²,

Chang³

et al. 2022

Plasma Phys. Control. Fusion

Self Cite

View full text Add to dashboard Cite

Experiments on fusion plasmas produce high-dimensional data time series with ever increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article we introduce the DELTA framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, DELTA allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describe the modular and expandable software architecture of DELTA and present performance benchmarks of individual components as well as of an example workflows. Focusing on a streaming analysis workflow where Electron cyclotron emission imaging (ECEi) data measured at KSTAR on NERSC's supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. At NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes in under 5 minutes. We further discuss how DELTA uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. By providing session leaders and physics operators results of higher order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

show abstract

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Cited by 6 publications

References 32 publications

2022 Review of Data-Driven Plasma Science

2022 Review of Data-Driven Plasma Science

Near real-time streaming analysis of big fusion data

Near real-time streaming analysis of big fusion data

Contact Info

Product

Resources

About