Capturing missing physics in climate model parameterizations using neural differential equations

Abstract

We explore how neural differential equations (NDEs) may be trained on highly resolved fluid-dynamical models of unresolved scales providing an ideal framework for data-driven parameterizations in climate models. NDEs overcome some of the limitations of traditional neural networks (NNs) in fluid dynamical applications in that they can readily incorporate conservation laws and boundary conditions and are stable when integrated over time. We advocate a method that employs a 'residual' approach, in which the NN is used to improve upon an existing parameterization through the representation of residual fluxes which are not captured by the base parameterization. This reduces the amount of training required and providing a method for capturing up-gradient and nonlocal fluxes. As an illustrative example, we consider the parameterization of free convection of the oceanic boundary layer triggered by buoyancy loss at the surface. We demonstrate that a simple parameterization of the process -convective adjustment -can be improved upon by training a NDE against highly resolved explicit models, to capture entrainment fluxes at the base of the well-mixed layer, fluxes that convective adjustment itself cannot represent. The augmented parameterization outperforms existing commonly used parameterizations such as the K-Profile Parameterization (KPP). We showcase that the NDE performs well independent of the time-stepper and that an online training approach using differentiable simulation via the Julia scientific machine learning software stack improves accuracy by an order-of-magnitude. We conclude that NDEs provide an exciting route forward to the development of representations of sub-grid-scale processes for climate science, opening up myriad new opportunities.

Deep Learned Process Parameterizations Provide Better Representations of Turbulent Heat Fluxes in Hydrologic Models

“…Recurrent layers such as LSTMs have been dominant in the timeseries domain. More sophisticated architectures such as neural ordinary differential equations (Ramadhan et al, 2020) or those discovered through neural architecture search (Geng & Wang, 2020) are bound to be both more efficient and interpretable than our dense networks. The opportunities for incorporating and learning from ML-based models into the hydrologic sciences are virtually untapped.…”

“…This approach has been adopted recently by Brenowitz and Bretherton (2018) as well as Rasp et al (2018) for parameterizing sub-gridcell scale processes, such as cloud convection, in atmospheric circulation models. Similarly, in oceanography, neural networks have been used to parameterize the turbulent vertical mixing in the ocean surface (Ramadhan et al, 2020).…”

Artificial Intelligence for Quantitative Modeling in Drug Discovery and Development: An Innovation and Quality Consortium Perspective on Use Cases and Best Practices

“…In fact, XAI requires combining human intuition and systematic thinking with the ability of ML to process vast amounts of data. Scientific machine learning is one such approach where domain knowledge is coupled to flexible ML techniques in the initial framework design (also termed glass‐box) to improve both accuracy and explainability, 67 but it requires more expertise to create. The transparency of ML algorithms is closely linked to their explainability, and by providing clarity to the model's internal workings it can instill greater confidence among stakeholders in the reliability and validity of the model's outputs 68 .…”

Parameterizing Vertical Mixing Coefficients in the Ocean Surface Boundary Layer Using Neural Networks

“…Studies show that neural networks can be used in idealized model configurations, and recently, the use of machine learning has emerged in realistic GCMs. Artificial neural networks (ANNs) have been shown to improve sub‐grid momentum transport in atmospheric models (Yuval & O'Gorman, 2023), predict precipitation (Shamekh et al., 2023) and fluxes (Shamekh & Gentine, 2023), while in ocean models they have been used to improve the parameterization of free convection (Ramadhan et al., 2023). Liang et al.…”