Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

Ejarque, Jorge; Badía, Rosa M.; Albertin, Loïc; Aloisio, Giovanni; Baglione, Enrico; Becerra, Yolanda; Boschert, Stefan; Berlin, Julian R.; D'Anca, Alessandro; Elia, Donatello; Exertier, François; Fiore, Sandro; Flich, Jose; Folch, Arnau; Gibbons, Steven J.; Koldunov, Nikolay; Lordan, Francesc; Lorito, Stefano; Løvholt, Finn; Macías, Jorge; Marozzo, Fabrizio; Michelini, Alberto; Monterrubio-Velasco, Marisol; Pienkowska, Marta; Puente, Josep de la; Queralt, Anna; Quintana–Ort́ı, Enrique S.; Rodriguez, J. E.; Romano, Fabrizio; Rossi, Riccardo; Rybicki, Jedrzej; Kupczyk, Miroslaw; Selva, Jacopo; Talia, Domenico; Tonini, Roberto; Trunfio, Paolo; Volp, Manuela

doi:10.1016/j.future.2022.04.014

Cited by 26 publications

(13 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main goal of eFlows4HPC is to deliver a workflow software stack and an additional set of services that enable the integration of HPC simulations and modeling with high-performance data analytics in scientific and industrial applications, with urgent computing workflows as relevant use cases. 2 Developing urgent computing workflows for natural hazards such as tsunamis and earthquakes involves deploying advanced tools and complex tasks to ultimately bring them to an operational level. Urgent computing workflows take advantage of the eFlows4HPC software stack to improve technology and provide fast solutions for mitigating the effects of potentially catastrophic tsunamis and earthquakes.…”

Section: Significant Projectsmentioning

confidence: 99%

Urgent Computing for Protecting People From Natural Disasters

Talia

Trunfio

2023

Computer

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Significant Projectsmentioning

confidence: 99%

Urgent Computing for Protecting People From Natural Disasters

Talia

Trunfio

2023

Computer

Self Cite

View full text Add to dashboard Cite

show abstract

“…AI-integrated workflows are emerging as a powerful tool across many scientific domains, with many use patterns [37,38]. Following the lexicon of Jha et al [37], there are at least "ML-in-HPC" applications where AI-based software are used inside conventional HPC applications (e.g., surrogates for expensive computational routines [39]) and "ML-out-HPC" where machine learning controls the execution of the application (e.g., steering a workflow via active learning [40][41][42]).…”

Section: Ai-integrated Workflowsmentioning

confidence: 99%

Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources

Ward¹,

Pauloski²,

Hayot‐Sasson³

et al. 2023

Preprint

View full text Add to dashboard Cite

Applications that fuse machine learning and simulation can benefit from the use of multiple computing resources, with, for example, simulation codes running on highly parallel supercomputers and AI training and inference tasks on specialized accelerators. Here, we present our experiences deploying two AIguided simulation workflows across such heterogeneous systems. A unique aspect of our approach is our use of cloud-hosted management services to manage challenging aspects of crossresource authentication and authorization, function-as-a-service (FaaS) function invocation, and data transfer. We show that these methods can achieve performance parity with systems that rely on direct connection between resources. We achieve parity by integrating the FaaS system and data transfer capabilities with a system that passes data by reference among managers and workers, and a user-configurable steering algorithm to hide data transfer latencies. We anticipate that this ease of use can enable routine use of heterogeneous resources in computational science.

show abstract

“…Climate variables are the main drivers that contribute to the formation and strengthening of TCs during their lifetime, and they were retrieved from the Copernicus Climate Change Service ERA5 reanalysis data sets. ERA5 reanalysis combines global numerical weather predictions with newly available observations in an optimal way to produce consistent estimates of the state of the atmosphere (ECMWF, 2020b). In this study, MSLP [Pa] (msl), 10 m wind gust since previous post-processing [ms −1 ] (fg10) and the instantaneous 10 m wind gust [ms −1 ] (i10fg) were gathered from the ERA5 reanalysis on single levels (Hersbach et al, 2023b), whereas the relative vorticity at 850 mb [s −1 ] (vo850) and the temperature at 300 and 500 mb [K] (t300 and t500, respectively) were collected from the ERA5 reanalysis on the pressure levels data set (Hersbach et al, 2023a).…”

Section: Era5 Reanalysismentioning

confidence: 99%

An Ensemble Machine Learning Approach for Tropical Cyclone Localization and Tracking From ERA5 Reanalysis Data

Accarino,

Donno,

Immorlano

et al. 2023

Earth and Space Science

Self Cite

View full text Add to dashboard Cite

Tropical Cyclones (TCs) are counted among the most destructive phenomena that can be found in nature. Every year, globally an average of 90 TCs occur over tropical waters, and global warming is making them stronger and more destructive. The accurate localization and tracking of such phenomena have become a relevant and interesting area of research in weather and climate science. Traditionally, TCs have been identified in large climate data sets through the use of deterministic tracking schemes that rely on subjective thresholds. This study presents a Machine Learning (ML) ensemble approach for locating TCs center coordinates. The ensemble combines TCs center estimates of different ML models that agree about the presence of a TC in input data. ERA5 reanalysis data was used for model training and testing jointly with the International Best Track Archive for Climate Stewardship (IBTrACS) records. Compared to single models estimates, the ML ensemble approach was able to improve TCs localization in terms of Euclidean Distance with respect to the observed TCs locations from IBTrACS. Moreover, a hybrid tracking scheme was defined: starting from the individual TC center locations detected by the ML ensemble approach, a deterministic tracking algorithm was used for reconstructing TC trajectories. The hybrid tracking scheme was then compared with four deterministic trackers reported in literature, achieving a Probability of Detection and a False Alarm Rate of 71.49% and 23%, respectively, over 40 years of reanalysis data.

show abstract

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

Cited by 26 publications

References 55 publications

Urgent Computing for Protecting People From Natural Disasters

Urgent Computing for Protecting People From Natural Disasters

Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources

An Ensemble Machine Learning Approach for Tropical Cyclone Localization and Tracking From ERA5 Reanalysis Data

Contact Info

Product

Resources

About