No abstract

To make weather and climate models computationally affordable, small-scale processes are usually represented in terms of the large-scale, explicitly resolved processes using physics-based/ semi-empirical parameterization schemes. Another approach, computationally more demanding but often more accurate, is super-parameterization (SP). SP involves integrating the equations of small-scale processes on high-resolution grids embedded within the low-resolution grid of large-scale processes. Recently, studies have used machine learning (ML) to develop data-driven parameterization (DD-P) schemes. Here, we propose a new approach, data-driven SP (DD-SP), in which the equations of the small-scale processes are integrated data-drivenly (thus inexpensively) using ML methods such as recurrent neural networks. Employing multiscale Lorenz 96 systems as the testbed, we compare the cost and accuracy (in terms of both short-term prediction and long-term statistics) of parameterized low-resolution (PLR) SP, DD-P, and DD-SP models. We show that with the same computational cost, DD-SP substantially outperforms PLR and is more accurate than DD-P, particularly when scale separation is lacking. DD-SP is much cheaper than SP, yet its accuracy is the same in reproducing long-term statistics (climate prediction) and often comparable in short-term forecasting (weather prediction). We also investigate generalization: when models trained on data from one system are applied to a more chaotic system, we find that models often do not generalize, particularly when short-term prediction accuracies are examined. However, we show that transfer learning, which involves retraining the data-driven model with a small amount of data from the new system, significantly improves generalization. Potential applications of DD-SP and transfer learning in climate/weather modeling are discussed. Plain Language Summary The weather/climate system involves intertwined physical processes acting on scales from centimeters (or even smaller) to tens of thousands of kilometers. Most weather/climate models used in practice include parameterization schemes that relate small-scale processes, which are not explicitly resolved (due to coarse spatiotemporal resolution), to large-scale processes that are resolved. Recently, studies have explored using machine learning for data-driven parameterization (DD-P) of small-scale (subgrid) processes. Here, we first introduce a novel way to leverage recent advances in deep learning to improve the modeling of subgrid processes. In this approach, called data-driven super-parameterization (DD-SP), deep learning is used for fast, data-driven integration of equations of small-scale processes, while other equations are integrated using conventional numerical methods. Employing a relatively simple chaotic system, we show the advantages of DD-SP over DD-P and conventional parameterizations. Second, we examine how these data-driven models generalize (extrapolate) from one system to other (e.g., more chaotic) systems. We demonstrate that these mod...

No abstract

Developing data-driven subgrid-scale (SGS) models for large eddy simulations (LES) has received substantial attention recently. Despite some success, particularly in a priori (offline) tests, challenges have been identified that include numerical instabilities in a posteriori (online) tests and generalization (i.e., extrapolation) of trained data-driven SGS models, for example to higher Reynolds numbers. Here, using the stochastically forced Burgers turbulence as the test-bed, we show that deep neural networks trained using properly pre-conditioned (augmented) data yield stable and accurate a posteriori LES models. Furthermore, we show that transfer learning enables accurate/stable generalization to a flow with 10× higher Reynolds number.

Transfer learning (TL), which enables neural networks (NNs) to generalize out-of-distribution via targeted re-training, is becoming a powerful tool in scientific machine learning (ML) applications such as weather/climate prediction and turbulence modeling. Effective TL requires knowing 1) how to re-train NNs? and 2) what physics are learned during TL? Here, we present novel analyses and a framework addressing (1)-(2) for a broad range of multi-scale, nonlinear, dynamical systems. Our approach combines spectral (e.g., Fourier) analyses of such systems with spectral analyses of convolutional NNs, revealing physical connections between the systems and what the NN learns (a combination of low-, high-, band-pass filters and Gabor filters). Integrating these analyses, we introduce a general framework that identifies the best re-training procedure for a given problem based on physics and NN theory. As test case, we explain the physics of TL in subgrid-scale modeling of several setups of 2D turbulence. Furthermore, these analyses show that in these cases, the shallowest convolution layers are the best to re-train, which is consistent with our physics-guided framework but is against the common wisdom guiding TL in the ML literature. Our work provides a new avenue for optimal and explainable TL, and a step toward fully explainable NNs, for wide-ranging applications in science and engineering, such as climate change modeling.

Transfer learning (TL) is becoming a powerful tool in scientific applications of neural networks (NNs), such as weather/climate prediction and turbulence modeling. TL enables out-of-distribution generalization (e.g., extrapolation in parameters) and effective blending of disparate training sets (e.g., simulations and observations). In TL, selected layers of a NN, already trained for a base system, are re-trained using a small dataset from a target system. For effective TL, we need to know 1) what are the best layers to re-train? and 2) what physics are learned during TL? Here, we present novel analyses and a new framework to address (1)-( 2) for a broad range of multi-scale, nonlinear systems. Our approach combines spectral analyses of the systems' data with spectral analyses of convolutional NN's activations and kernels, explaining the inner-workings of TL in terms of the system's nonlinear physics. Using subgrid-scale modeling of several setups of 2D turbulence as test cases, we show that the learned kernels are combinations of low-, band-, and high-pass filters, and that TL learns new filters whose nature is consistent with the spectral differences of base and target systems. We also find the shallowest layers are the best to re-train in these cases, which is against the common wisdom guiding TL in machine learning literature. Our framework identifies the best layer(s) to re-train beforehand, based on physics and NN theory. Together, these analyses explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.

No abstract

<p>In large eddy simulations (LES), the subgrid-scale effects are modeled by physics-based or data-driven methods. This work develops a convolutional neural network (CNN) to model the subgrid-scale effects of a two-dimensional turbulent flow. The model is able to capture both the inter-scale forward energy transfer and backscatter in both a priori and a posteriori analyses. The LES-CNN model outperforms the physics-based eddy-viscosity models and the previous proposed local artificial neural network (ANN) models in both short-term prediction and long-term statistics. Transfer learning is implemented to generalize the method for turbulence modeling at higher Reynolds numbers. Encoder-decoder network architecture is proposed to generalize the model to a higher computational grid resolution.</p>