Differentiable modeling to unify machine learning and physical models and advance Geosciences

Abstract

<p>Process-Based Modeling (PBM) and Machine Learning (ML) are often perceived as distinct paradigms in the geosciences. Here we present differentiable geoscientific modeling as a powerful pathway toward dissolving the perceived barrier between them and ushering in a paradigm shift. For decades, PBM offered benefits in interpretability and physical consistency but struggled to efficiently leverage large datasets. ML methods, especially deep networks, presented strong predictive skills yet lacked the ability to answer specific scientific questions. While various methods have been proposed for ML-physics integration, an important underlying theme  — differentiable modeling — is not sufficiently recognized. Here we outline the concepts, applicability, and significance of differentiable geoscientific modeling (DG). “Differentiable” refers to accurately and efficiently calculating gradients with respect to model variables, critically enabling the learning of high-dimensional unknown relationships. DG refers to a range of methods connecting varying amounts of prior knowledge to neural networks and training them together, capturing a different scope than physics-guided machine learning and emphasizing first principles. In this talk we provide examples of DG in global hydrology, ecosystem modeling, water quality simulations, etc. Preliminary evidence suggests DG offers better interpretability and causality than ML, improved generalizability and extrapolation capability, and strong potential for knowledge discovery, while approaching the performance of purely data-driven ML. DG models require less training data while scaling favorably in performance and efficiency with increasing amounts of data. With DG, geoscientists may be better able to frame and investigate questions, test hypotheses, and discover unrecognized linkages. </p>

Evaluating a global soil moisture dataset from a multitask model (GSM3 v1.0) with potential applications for crop threats

“…Even though that model does not simulate the physical quantity of soil moisture, it could be modified to have a module that does. However, to obtain suitable parameters on the global scale and improve the physical processes, we think adding differentiable programming to the model will give it the adaptive capability to learn from big data (Feng et al, 2022;Shen et al, 2023;Aboelyazeed et al, 2022;Bindas et al, 2022). It is possible that such a model may generalize better than LSTM over long distances due to the imposed physical constraints.…”

“…Typically, for many hydrologic applications (Fang et al, 2022;Liu et al, 2022a;Rahmani et al, 2021a), a spatial test is a tougher test than a temporal test for fully data-driven models, showing the strong impacts of spatial heterogeneity. This could either mean the inputs of the model do not completely describe the problem or that there are not enough sites in space with different combinations of input attributes for the model to fully resolve their impacts.…”

“…Component (iv) appears as a large difference between training and testing metrics. It is worthwhile to note that LSTM models typically (although not always) perform better for each site when given data from more numerous or more diverse sites due to a "data synergy" effect (Fang et al, 2022).…”

“…Due to LSTM's strong ability to fit to data, it can serve as a probe for process complexity (Liu et al, 2022a;Feng et al, 2022Feng et al, , 2020Tsai et al, 2021): those sites that LSTM cannot adequately capture may contain complicated processes that are not well described by the inputs. The factorial importance analysis indicates that slope aspect, average soil moisture, and surface solar radiation downwards are the top three factors that influence the multitask LSTM model's R in the temporal test (Fig.…”

Train, Inform, Borrow, or Combine? Approaches to Process‐Guided Deep Learning for Groundwater‐Influenced Stream Temperature Prediction

“…Additional custom loss terms would be possible and could make use of other data, such as observed groundwater temperatures and levels, if a differentiable model were used that represented those intermediate variables within process‐based equations (Shen et al., 2023). In the DRB, there were only two sites with groundwater wells with daily water temperature observations (all occurring within the test partition) and only 24 wells with more than 20 discrete groundwater temperature observations.…”