Towards Interactive, Reproducible Analytics at Scale on HPC Systems

Cholia, Shreyas; Heagy, Lindsey J.; Henderson, Matthew; Paine, Drew; Hays, Jon; Bianchi, Ludovico; Ghoshal, Devarshi; Pérez, Fernando; Ramakrishnan, Lavanya

doi:10.1109/urgenthpc51945.2020.00011

Cited by 3 publications

(2 citation statements)

References 21 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It utilizes Dask, which is a flexible library for parallel computing in Python and allows for dynamic task scheduling for optimizing computational workloads as well as handling large datasets. 43,44 Furthermore, it uses Apache Parquet file format, which is an efficient, structured, and column-oriented binary format capable of file size reduction and efficient reading of the desired stored values without the need to load the entire file into memory. 45 More concretely, to address the above-mentioned challenges, PARAMOUNT has been developed with the capability of assembling the data matrix Y for a list of variables of interest.…”

Section: Pod Formulation In Paramountmentioning

confidence: 99%

Combustion dynamics analysis of a pressurized airblast swirl burner using proper orthogonal decomposition

Ghasemi,

Christou,

Kok

et al. 2023

International Journal of Spray and Combustion Dynamics

View full text Add to dashboard Cite

Jet fuel-fired combustors in aero gas turbine engines have switched to lean burn to decrease nitric oxide emissions in recent years as a result of strict emission regulations. Lean operating conditions, however, exhibit a heightened sensitivity to thermoacoustic instabilities and such burners require careful consideration in design and operation. Similar to natural gas-fired combustors, they exhibit thermoacoustic instabilities, but the characteristics are more complex and less well-studied. This paper presents a numerical investigation of an airblast jet fuel swirl burner operating with preheated air at lean pressurized conditions. In order to understand the acoustic characteristics of the in-house designed burner (Magister UT burner), detached eddy simulations are performed at relevant aero engine conditions. Simulation results are then analyzed by means of our internally developed parallel modal analysis package, PARAMOUNT, to perform proper orthogonal decomposition (POD) on large datasets. The resulting modes are inspected to highlight flow features of interest and their associated acoustic frequencies at unforced conditions. Single frequency acoustic forcing is employed to study the acoustic response of the burner to perturbations at similar frequencies to its precessing vortex core. We show that parallel computation of POD modes is a viable tool to investigate the main flow features of swirl burners and is suitable for highlighting the important acoustic frequencies without the need to employ fully compressible computational fluid dynamics solvers. Additionally, the analysis method reveals the ways in which various flow structures correlate with each other and how external perturbations modify them.

show abstract

Section: Pod Formulation In Paramountmentioning

confidence: 99%

Combustion dynamics analysis of a pressurized airblast swirl burner using proper orthogonal decomposition

Ghasemi,

Christou,

Kok

et al. 2023

International Journal of Spray and Combustion Dynamics

View full text Add to dashboard Cite

show abstract

“…In addition, it provides a user-friendly platform to document the software while ensuring reproducibility by combining the data, code and software environment. As Jupyter notebook offers the core, JupyterHub expands the frameworks and brings flexibility to the user group [30]. JupyterHub provides access control and authentication, scalability with support for container and HPC technology, and it is portable from the cloud to a local machine.…”

Section: Notebook and Gitmentioning

confidence: 99%

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

et al. 2022

View full text Add to dashboard Cite

Machine learning (ML) applications in weather and climate are gaining momentum as big data and the immense increase in High-performance computing (HPC) power are paving the way. Ensuring FAIR data and reproducible ML practices are significant challenges for Earth system researchers. Even though the FAIR principle is well known to many scientists, research communities are slow to adopt them. Canonical Workflow Framework for Research (CWFR) provides a platform to ensure the FAIRness and reproducibility of these practices without overwhelming researchers. This conceptual paper envisions a holistic CWFR approach towards ML applications in weather and climate, focusing on HPC and big data. Specifically, we discuss Fair Digital Object (FDO) and Research Object (RO) in the DeepRain project to achieve granular reproducibility. DeepRain is a project that aims to improve precipitation forecast in Germany by using ML. Our concept envisages the raster datacube to provide data harmonization and fast and scalable data access. We suggest the Juypter notebook as a single reproducible experiment. In addition, we envision JuypterHub as a scalable and distributed central platform that connects all these elements and the HPC resources to the researchers via an easy-to-use graphical interface.

show abstract

Asynchronous Multi-Level Checkpointing: An Enabler of Reproducibility using Checkpoint History Analytics

Assogba,

Nicolae,

Van Dam

et al. 2023

Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analys

View full text Add to dashboard Cite

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

Cited by 3 publications

References 21 publications

Combustion dynamics analysis of a pressurized airblast swirl burner using proper orthogonal decomposition

Combustion dynamics analysis of a pressurized airblast swirl burner using proper orthogonal decomposition

HPC-oriented Canonical Workflows for Machine Learning Applications in Climate and Weather Prediction

Asynchronous Multi-Level Checkpointing: An Enabler of Reproducibility using Checkpoint History Analytics

Contact Info

Product

Resources

About