Stream processing for near real-time scientific data analysis

Choi, Jong Youl; Kurç, Tahsin; Logan, Jeremy; Wolf, Matthew; Suchyta, E.; Kress, James; Pugmire, David; Podhorszki, Norbert; Byun, Eun-Kyu; Ainsworth, Mark; Parashar, Manish; Klasky, Scott

doi:10.1109/nysds.2016.7747804

Cited by 7 publications

(6 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, reduction methods were applied to the gas puff imaging (GPI) diagnostic prior to remote transfer in order to send only regions that contained large deviations from the average (potentially containing blob regions). 2 These data transformations can include physics priors (such as the GPI example to send only possible blob regions) or can be agnostic to the underlying physics such as compression or indexing algorithms. 1 ADIOS already has several compression algorithms included in the framework, from lossless compression techniques such as SZIP (Ref.…”

Section: Iiic Reduction Methodsmentioning

confidence: 99%

“…The global nature of the ITER project along with its projected approximately petabyte-per-day data generation presents a not only unique challenge but also an opportunity for the fusion community to rethink, optimize, and enhance our scientific discovery process. Recognizing this, collaborative research 1,2 with computational scientists was undertaken over the past several years toward building a framework for large-scale data movement across widearea networks (WANs) to enable global near-real-time analysis of data from remote experimental devices. This would broaden the available computational resources for analysis/simulation and increase the number of researchers actively participating in experiments.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Churchill

Chang

Choi

et al. 2021

Fusion Science and Technology

View full text Add to dashboard Cite

The global nature of the ITER project along with its projected approximately petabyte-per-day data generation presents not only a unique challenge but also an opportunity for the fusion community to rethink, optimize, and enhance our scientific discovery process. Recognizing this, collaborative research with computational scientists was undertaken over the past several years to create a framework for large-scale data movement across wide-area networks to enable global near-real-time analysis of fusion data. This would broaden the available computational resources for analysis/simulation and increase the number of researchers actively participating in experiments.An official demonstration of this framework for fast, large data transfer and real-time analysis was carried out between the KSTAR tokamak in Daejeon, Korea, and Princeton Plasma Physics Laboratory (PPPL) in Princeton, New Jersey. Streaming large data transfer, with near-real-time movie creation and analysis of the KSTAR electron cyclotron emission imaging data, was performed using the Adaptable Input Output (I/O) System (ADIOS) framework, and comparisons were made at PPPL with simulation results from the XGC1 code. These demonstrations were made possible utilizing an optimized network configuration at PPPL, which achieved over 8.8 Gbps (88% utilization) in throughput tests from the National Fusion Research Institute to PPPL. This demonstration showed the feasibility for large-scale data analysis of KSTAR data and provides a nascent framework to enable use of globally distributed computational and personnel resources in pursuit of scientific knowledge from the ITER experiment.

show abstract

Section: Iiic Reduction Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Churchill

Chang

Choi

et al. 2021

Fusion Science and Technology

View full text Add to dashboard Cite

show abstract

“…Magnetic islands can thus flatten the electron temperature profile once they are sufficiently large. The footprint of this effect in ECEi data are so-called radial phase inversion structures, quadrupole-like coherent patches in the local T e measurements Choi et al [2016].…”

Section: Automated Image Analysismentioning

confidence: 99%

“…In this paper we are presenting a streaming data analysis framework that aims to connect fusion experiments with remote HPC resources Choi et al [2016], Ralph , , Churchill et al [2021]. The streaming paradigm implemented by this framework allows the big-and fast data generated by fusion experiments to be seamlessly analyzed using supercomputers as they are generated.…”

Section: Introductionmentioning

confidence: 99%

Near real-time streaming analysis of big fusion data

Kube,

Churchill,

Chang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

While experiments on fusion plasmas produce high-dimensional data time series with ever increasing magnitude and velocity, data analysis has been lagging behind this development. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a highperformance compute center, Delta allows to perform demanding data analysis tasks in between plasma pulses. This article describe the modular and expandable software architecture of Delta and presents performance benchmarks of its individual components as well as of entire workflows. Our focus is on the streaming analysis of ECEi data measured at KSTAR on NERSCs supercomputers and we routinely achieve data transfer rates of about 500 MByte per second. We show that a demanding turbulence analysis workload can be distributed among multiple GPUs and executes in under 5 minutes. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based realtime data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. By providing session leaders and physics operators results of higher order data analysis using live visualization they may monitor the evolution of a long-pulse discharge in near real-time and may make more informed decision on how to configure the machine for the next shot.

show abstract

“…In this paper, we present a streaming data analysis framework that aims to connect fusion experiments with remote HPC resources [2,6,19,32]. The streaming paradigm implemented by this framework allows the data generated by fusion experiments to be seamlessly analyzed using supercomputers as they are generated.…”

Section: Introductionmentioning

confidence: 99%

Near real-time streaming analysis of big fusion data

Kube¹,

Churchill²,

Chang³

et al. 2022

Plasma Phys. Control. Fusion

View full text Add to dashboard Cite

Experiments on fusion plasmas produce high-dimensional data time series with ever increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article we introduce the DELTA framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, DELTA allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describe the modular and expandable software architecture of DELTA and present performance benchmarks of individual components as well as of an example workflows. Focusing on a streaming analysis workflow where Electron cyclotron emission imaging (ECEi) data measured at KSTAR on NERSC's supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. At NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes in under 5 minutes. We further discuss how DELTA uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. By providing session leaders and physics operators results of higher order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

show abstract

Stream processing for near real-time scientific data analysis

Cited by 7 publications

References 21 publications

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Near real-time streaming analysis of big fusion data

Near real-time streaming analysis of big fusion data

Contact Info

Product

Resources

About