Technical note: Data assimilation and autoregression for using near-real-time streamflow observations in long short-term memory networks

Abstract

Abstract. Ingesting near-real-time observation data is a critical component of many operational hydrological forecasting systems. In this paper, we compare two strategies for ingesting near-real-time streamflow observations into long short-term memory (LSTM) rainfall–runoff models: autoregression (a forward method) and variational data assimilation. Autoregression is both more accurate and more computationally efficient than data assimilation. Autoregression is sensitive to missing data, however an appropriate (and simple) training strategy mitigates this problem. We introduce a data assimilation procedure for recurrent deep learning models that uses backpropagation to make the state updates.

Optimal Postprocessing Strategies With LSTM for Global Streamflow Prediction in Ungauged Basins

“…Based on this, the dimensional‐reduction operator of the CNN was employed in FDC to reduce overfitting, from which CNN‐DI (FDC) and CNN‐DI (ISIMIP, FDC) can be formed. Among six sub‐experiments, AR (1) and AR (3) are autoregressive (AR) methods suitable for hydrological simulations in data‐rich region (G. S. Nearing et al., 2022), while LSTM, CNN‐DI (FDC), DI (ISIMIP), and CNN‐DI (ISIMIP, FDC) are suitable for hydrological simulations in ungauged basins. Autoregressive methods are not placed here to participate in experimental comparisons, but rather to highlight the huge gap between current baseline LSTM and autoregressive methods, and show that GHM‐forced LSTM are the best way to close this gap.…”

Machine Learning for Bayesian Experimental Design in the Subsurface

“…LSTM is a neural network architecture that is well-suited for modeling timeseries data, and may be an excellent candidate for modeling dynamical systems such as watersheds (Kratzert et al, 2019b). Lees et al (2022) and Nearing et al (2022) investigated the information captured by the LSTM state vector and compared two approaches for ingesting near-real-time streamflow observations for rainfall-runoff modeling. Li et al (2021) suggested using Bayesian LSTM with stochastic variational inference to estimate model uncertainty in process-based hydrological models.…”

“…Data fusion is the process of combining information from different sources (e.g., sensors, databases, human expertise), and is a key method for handling imperfect raw data to obtain useful and accurate information (Meng et al, 2020). The process of combining observations into computer models is also known as data assimilation, and it is widely used in numerous fields, including atmospheric prediction, seismology, energy and environmental applications (Lavin et al, 2021;Nearing et al, 2018aNearing et al, , 2022Tso et al, 2020).…”

Can machine learning accelerate process understanding and decision‐relevant predictions of river water quality?

“…Many ML studies use readily available off‐the‐shelf products such as the catchment attributes and meteorology for large‐sample studies (CAMELS; Addor et al, 2017), which may impair their extensibility to practical, realistic applications where such data may not be available. Assimilation of new data into models using methods such as ensemble Kalman filters and autoregression (Brajard et al, 2020; Nearing, Klotz, et al, 2021; Zwart et al, 2021), and the use of integrated datasets tailored for the problem can improve prediction outcomes. Software that synthesize data for on‐demand queries such as brokering‐based tools (Horsburgh et al, 2016; Varadharajan et al, 2022), and methods to streamline quality control and outlier detection, gap‐fill, downscale observations, and determine parameters for process models (Bennett & Nijssen, 2021; Campbell et al, 2013; Hill & Minsker, 2010; Leigh et al, 2019; Mital et al, 2020; Russo et al, 2020) would ideally be integrated into ML workflows in parallel with advances in modelling approaches.…”