Hydrological Concept Formation inside Long Short-Term Memory (LSTM) networks

Abstract

Abstract. Neural networks have been shown to be extremely effective rainfall-runoff models, where the river discharge is predicted from meteorological inputs. However, the question remains, what have these models learned? Is it possible to extract information about the learned relationships that map inputs to outputs? And do these mappings represent known hydrological concepts? Small-scale experiments have demonstrated that the internal states of Long Short-Term Memory Networks (LSTMs), a particular neural network architecture predisposed to hydrological modelling, can be interpreted. By extracting the tensors which represent the learned translation from inputs (precipitation, temperature) to outputs (discharge), this research seeks to understand what information the LSTM captures about the hydrological system. We assess the hypothesis that the LSTM replicates real-world processes and that we can extract information about these processes from the internal states of the LSTM. We examine the cell-state vector, which represents the memory of the LSTM, and explore the ways in which the LSTM learns to reproduce stores of water, such as soil moisture and snow cover. We use a simple regression approach to map the LSTM state-vector to our target stores (soil moisture and snow). Good correlations (R2 > 0.8) between the probe outputs and the target variables of interest provide evidence that the LSTM contains information that reflects known hydrological processes comparable with the concept of variable-capacity soil moisture stores. The implications of this study are threefold: 1) LSTMs reproduce known hydrological processes. 2) While conceptual models have theoretical assumptions embedded in the model a priori, the LSTM derives these from the data. These learned representations are interpretable by scientists. 3) LSTMs can be used to gain an estimate of intermediate stores of water such as soil moisture. While machine learning interpretability is still a nascent field, and our approach reflects a simple technique for exploring what the model has learned, the results are robust to different initial conditions and to a variety of benchmarking experiments. We therefore argue that deep learning approaches can be used to advance our scientific goals as well as our predictive goals.

Assessing the Physical Realism of Deep Learning Hydrologic Model Projections Under Climate Change

“…An analysis of state variables from the process models (not shown) confirms this is due to more precipitation as rain in the cold season, earlier snowmelt, and increased evapotranspiration triggered by warming. The CV-LSTM captures the same basic pattern despite not having explicit representations of these processes in its architecture; these processes are instead represented by the internal states of the network (Lees et al, 2021).…”

“…Recent work has shown that LSTMs trained to hundreds of basins can predict extreme events with more accuracy than process models, even when the largest extremes are withheld during training (Frame, Kratzert, Klotz, et al, 2021). In addition, Lees et al (2021) showed that the cell states of globally trained LSTMs used to predict streamflow strongly correlated with latent hydrologic states, such as soil moisture storage and snowpack. These results suggest that DL hydrologic models are learning fundamental processes from the data, a concept further bolstered by the fact that LSTMs trained to many basins produce better out-of-sample streamflow predictions in a given basin than an LSTM only trained to that basin or a smaller subset (Gauch, Mai, & Lin, 2021;Kratzert, Klotz, Shalev, et al, 2019).…”

Differentiable, Learnable, Regionalized Process‐Based Models With Multiphysical Outputs can Approach State‐Of‐The‐Art Hydrologic Prediction Accuracy

“…For hydrologic modeling, it remains challenging to interpret what is being learned by LSTM, in part because it does not output physical states or fluxes. While it is possible to correlate LSTM's cell states to some physical states with regression approaches (Lees et al, 2021), the physical meaning of these cell states cannot be guaranteed explicitly, and we cannot run such regression tests before having access to the observations of those physical variables. The regression approach also does not allow us to freely ask questions about how the system functions.…”

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

“…Also in the rainfall‐runoff domain, Kratzert et al (2019) used integrated gradients (Sundararajan et al, 2017) to confirm a theory‐consistent influence of precipitation and air temperature on a NN state that correlated with snow water equivalent, building trust in the ability of such models to capture known physical processes. More recent work on the same subject used probes (e.g., linear regression models or stacked multi‐layer perceptrons) to connect LSTM cell states to the output for tracking the evolution of the LSTM during the training process and verifying whether the LSTM learned physically realistic mappings from inputs to outputs (Lees et al, 2021).…”