Towards HPC and Big Data Analytics Convergence: Design and Experimental Evaluation of a HPDA Framework for eScience at Scale

Abstract

Over the last two decades, scientific discovery has increasingly been driven by the large availability of data from a multitude of sources, including high-resolution simulations, observations and instruments, as well as an enormous network of sensors and edge components. In such a dynamic and growing landscape where data continue to expand, advances in Science have become intertwined with the capacity of analysis tools to effectively handle and extract valuable information from this ocean of data. In view of the exascale era of supercomputers that is rapidly approaching, it is of the utmost importance to design novel solutions that can take full advantage of the upcoming computing infrastructures. The convergence of High Performance Computing (HPC) and data-intensive analytics is key to delivering scalable High Performance Data Analytics (HPDA) solutions for scientific and engineering applications. The aim of this paper is threefold: reviewing some of the most relevant challenges towards HPDA at scale, presenting a HPDAenabled version of the Ophidia framework and validating the scalability of the proposed framework through an experimental performance evaluation carried out in the context of the Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE). The experimental results show that the proposed solution is capable of scaling over several thousand cores and hundreds of cluster nodes. The proposed work is a contribution in support of scientific large-scale applications along the wider convergence path of HPC and Big Data followed by the scientific research community.

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

“…The front-end server and the computing components are usually deployed on different nodes of the infrastructure. Furthermore, the number of Ophidia computing components can be scaled up, also dynamically, over multiple nodes of the infrastructure to address more intensive data analytics workloads [19]. To this end, the framework supports integration with different HPC scheduling systems.…”

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

“…In this respect, the climate community has been continuously pushing the boundaries to deploy and run model simulations at the highest resolution possible, exploiting cutting-edge supercomputing infrastructures [7]. The resulting output consists of large, complex and heterogeneous datasets that require proper solutions for management and knowledge extraction [8], and which can take advantage of data-oriented approaches from DA and AI fields.…”

An Ensemble Machine Learning Approach for Tropical Cyclone Localization and Tracking From ERA5 Reanalysis Data

“…Indeed, climate scientists, meteorological agencies and policy decision makers need to process and extract meaningful information from these huge data sets in a cost‐effective manner and in a reasonable amount of time (Sebestyén et al., 2021). In this context, High Performance Data analytics systems can address some of the issues and provide support for descriptive/statistical analysis of this large‐scale data (Elia et al., 2021). Nevertheless, in the last few years Machine Learning (ML) and Deep Learning (DL) algorithms became popular as data‐driven paradigms for supporting feature extraction from the vast amounts of scientific data currently available (Hey et al., 2020).…”