A crash-testing framework for predictive uncertainty assessment when forecasting high flows in an extrapolation context

Abstract

An increasing number of flood forecasting services assess and communicate the uncertainty associated with their forecasts. While obtaining reliable forecasts is a key issue, it is a challenging task, especially when forecasting high flows in an extrapolation context, i.e. when the event magnitude is larger than what was observed before. In this study, we present a crash-testing framework that evaluates the quality of hydrological forecasts in an extrapolation context. The experiment setup is based on i) a large set of catchments in France, ii) the GRP rainfall-runoff model designed for flood forecasting and used by the 5 French operational services and iii) an empirical hydrologic uncertainty processor designed to estimate conditional predictive uncertainty from the hydrological model residuals. The variants of the uncertainty processor used in this study differ in the data transformation they used (log, Box-Cox and log-sinh) to account for heteroscedasticity. Different data subsets were selected based on a preliminary event selection. Various aspects of the probabilistic performance of the variants of the hydrologic uncertainty processor, reliability, sharpness and overall quality, were evaluated. Overall, the results highlight the challenge of 10 uncertainty quantification when forecasting high flows. They show a significant drop in reliability when forecasting high flows in an extrapolation context and considerable variability among catchments and across lead times. The increase in statistical treatment complexity did not result in significant improvement, which suggests that a parsimonious and easily understandable data transformation such as the log transformation or the Box-Cox transformation with a parameter between 0.1 and 0.3 can be a reasonable choice for flood forecasting. uncertainty into its two main components is now common: the input uncertainty (mainly the meteorological forecast uncertainty) and the modelling uncertainty, as proposed by Krzysztofowicz (1999). More generally, the predictive uncertainty due to various sources may be explicitly modelled and propagated through the modelling chain, while the "remaining" uncertainty (from the other sources) may then be assessed by statistical post-processing. Modelling each source of uncertainty 5A first approach intends to model each source of uncertainty separately and to propagate these uncertainties through the modelling chain . The heteroscedasticity of the predictive uncertainty distribution results from the separate modelling of each source of uncertainty and from the statistical model specification. While this approach is promising, operational application can be hindered by the challenge of making the hydrological modelling uncertainty explicit, as pointed out by Salamon and Feyen (2009).

Improving the Reliability of Sub‐Seasonal Forecasts of High and Low Flows by Using a Flow‐Dependent Nonparametric Model

“…The MuTHRE‐FD model overcomes these problems. A common paradigm in forecasting is that reliability takes precedence over sharpness, because a prediction that is sharp but unreliable represents overconfidence (e.g., Berthet et al., 2020; Crochemore et al., 2016; Gneiting & Katzfuss, 2014). It follows that the large improvements in reliability of high flows obtained by the MuTHRE‐FD model are worth the sacrifice in sharpness.…”

Uncertainty Estimation with Deep Learning for Rainfall-Runoff Modelling

“…we lack standardized, open procedures for conducting comparative uncertainty estimation studies. Note that from the references above only Berthet et al (2020) focused on benchmarking uncertainty estimation strategies, and then only for assessing postprocessing approaches. We previously argued that data-based models provide a meaningful and general benchmark for testing hypotheses and models (Nearing and Gupta, 2015;Nearing et al, 2020b), and here we develop a set of data-based uncertainty estimation benchmarks built on a standard, publicly available, large-sample dataset that could be used as a baseline for future benchmarking studies.…”

Uncertainty Estimation with Deep Learning for Rainfall-Runoff Modelling

“…We struggled with finding suitable benchmarks for the DL uncertainty estimation approaches explored here. Ad hoc benchmarking and model intercomparison studies are common (e.g., Andréassian et al, 2009;Best et al, 2015;Kratzert et al, 2019b;Lane et al, 2019;Berthet et al, 2020;Nearing et al, 2018), and while the community has a (quickly growing) large-sample dataset for benchmarking hydrological models (Newman et al, 2017;Kratzert et al, 2019b), we lack standardized, open procedures for conducting comparative uncertainty estimation studies. Note that from the references above only Berthet et al (2020) focused on benchmarking uncertainty estimation strategies, and then only for assessing postprocessing approaches.…”

“…Ad hoc benchmarking and model intercomparison studies are common (e.g., Andréassian et al, 2009;Best et al, 2015;Kratzert et al, 2019b;Lane et al, 2019;Berthet et al, 2020;Nearing et al, 2018), and while the community has a (quickly growing) large-sample dataset for benchmarking hydrological models (Newman et al, 2017;Kratzert et al, 2019b), we lack standardized, open procedures for conducting comparative uncertainty estimation studies. Note that from the references above only Berthet et al (2020) focused on benchmarking uncertainty estimation strategies, and then only for assessing postprocessing approaches. We previously argued that data-based models provide a meaningful and general benchmark for testing hypotheses and models (Nearing and Gupta, 2015;Nearing et al, 2020b), and here we develop a set of data-based uncertainty estimation benchmarks built on a standard, publicly available, large-sample dataset that could be used as a baseline for future benchmarking studies.…”