Deep Uncertainty Quantification

Abstract

Weather forecasting is usually solved through numerical weather prediction (NWP), which can sometimes lead to unsatisfactory performance due to inappropriate setting of the initial states. In this paper, we design a data-driven method augmented by an effective information fusion mechanism to learn from historical data that incorporates prior knowledge from NWP. We cast the weather forecasting problem as an end-to-end deep learning problem and solve it by proposing a novel negative log-likelihood error (NLE) loss function. A notable advantage of our proposed method is that it simultaneously implements single-value forecasting and uncertainty quantification, which we refer to as deep uncertainty quantification (DUQ). Efficient deep ensemble strategies are also explored to further improve performance. This new approach was evaluated on a public dataset collected from weather stations in Beijing, China. Experimental results demonstrate that the proposed NLE loss significantly improves generalization compared to mean squared error (MSE) loss and mean absolute error (MAE) loss. Compared with NWP, this approach significantly improves accuracy by 47.76%, which is a state-of-the-art result on this benchmark dataset. The * Jie Lu and Tianrui Li are the correspondence authors of this paper. † Yu Zheng is also affiliated with Xidian University. preliminary version of the proposed method won 2nd place in an online competition for daily weather forecasting 1. CCS CONCEPTS • Applied computing → Environmental sciences.

Evaluation of machine learning techniques for forecast uncertainty quantification

“…A different approach estimates the forecast uncertainty directly from the data without requiring an a priori estimation of the forecast uncertainty, such as that provided by an ensemble of forecasts. D'Isanto and Polsterer (2018), Rasp and Lerch (2018), Wang et al (2018), Camporeale et al (2019), Barnes and Barnes (2021), and Veldkamp et al (2021) introduced NN architectures and training methodologies that directly quantify the uncertainty by performing the training over a large database of forecasts and their corresponding observations.…”

“…The key aspect is the definition of a loss function that should have a well-defined sensitivity to the state-dependent forecast uncertainty. Wang et al (2018) and Barnes and Barnes (2021) achieved this by defining a loss function inspired in the observation likelihood, whereas Camporeale (2018), D'Isanto andPolsterer (2018), Rasp and Lerch (2018), and Veldkamp et al (2021) defined a loss function based on the continuous ranked probability score and the Brier score. These studies concluded that this is a feasible approach for a cost-effective estimation of the forecast uncertainty for idealized and real-data cases.…”

A Survey on Uncertainty Quantification Methods for Deep Neural Networks: An Uncertainty Source Perspective

“…To summarize, the uncertainty of physical system modeling can come from the following source. First, the initial and boundary condition of the physical system is non-deterministic, and the system may be chaotic, i.e., in weather forecasting, a small perturbation can introduce a large deviation in future prediction [139]. Second, the inherent physical principle may not be perfectly known or the parameter of the governing equation may be stochastic, i.e., in an imperfect physical system, the conservation law of heat may be violated in a non-closed system [148].…”

A Multi-Factor Analysis of Forecasting Methods: A Study on the M4 Competition

“…In the meantime, and, as the M4 competition was running, many new methods were introduced. Novel approaches were presented ranging from linear differential equations [27], regularized regression [28], and of course deep learning approaches like Deep Clustering [29], Recurrent Neural Networks [30], Deep Ensembles [31] , Attention Networks [32], and Convolutional Networks [33]. All these methods were applied in multiple application domains ranging from weather forecasting to pandemics.…”