Potential of I/O Aware Workflows in Climate and Weather

Abstract

The efficient, convenient, and robust execution of data-driven workflows and enhanced data management are essential for productivity in scientific computing. In HPC, the concerns of storage and computing are traditionally separated and optimised independently from each other and the needs of the end-to-end user. However, in complex workflows, this is becoming problematic. These problems are particularly acute in climate and weather workflows, which as well as becoming increasingly complex and exploiting deep storage hierarchies, can involve multiple data centres. The key contributions of this paper are: 1) A sketch of a vision for an integrated data-driven approach, with a discussion of the associated challenges and implications, and 2) An architecture and roadmap consistent with this vision that would allow a seamless integration into current climate and weather workflows as it utilises versions of existing tools (ESDM, Cylc, XIOS, and DDN's IME). The vision proposed here is built on the belief that workflows composed of data, computing, and communication-intensive tasks should drive interfaces and hardware configurations to better support the programming models. When delivered, this work will increase the opportunity for smarter scheduling of computing by considering storage in heterogeneous storage systems. We illustrate the performance-impact on an example workload using a model built on measured performance data using ESDM at DKRZ.

A qualitative assessment of using ChatGPT as large language model for scientific workflow development

“…Of course, these only represent part of the full range of workflows in the natural sciences. It constitutes an exciting follow-up research question: how suitable ChatGPT and other generative LLMs are in other research contexts, such as climate research [ 33 ] and astronomy [ 32 ], and whether it is possible to identify categories or groups of domains that are particularly well (or poorly) supported. Furthermore, we examined only 2 workflow systems, Nextflow and Apache Airflow, leaving other alternatives, such as Snakemake, Taverna, and Pegasus, for future work.…”

“…Scientific workflows are widely used by diverse research communities, such as biomedicine [ 31 ], astronomy [ 32 ], climatology [ 33 ], and Earth observation [ 34 ], to manage the dataflow and distributed execution of complex analyses, simulations, and experiments. A scientific workflow comprises a series of interconnected computational steps, often with diverse patterns of dependencies, that define how to process and analyze data to reach a particular research objective.…”