In this paper, we comment on the recent two‐part paper by Goodwell and Kumar (2017a, https://doi.org/10.1002/2016WR020216, 2017b, https://doi.org/10.1002/2016WR020218) on quantifying three‐way interactions between variables using information theory. Their proposed method of partitioning interactions into unique, redundant, and synergistic information is valuable and has potential other applications in the field of water resources. We present an example to investigate the generality of their assumption that redundancy follows from dependency of the sources. In the broader context of information theoretical methods in the geosciences, we argue that implementation challenges stem mostly from issues that are intrinsic to learning patterns from limited data. These issues are only hidden by assumptions, but not absent, when using conventional correlation‐based methods. The flexibility of individually choosing assumptions in information theoretical methods gives them a myriad of potential applications in the study of complex systems.

Over the past two decades, the Bootstrap AGGregatING (bagging) method has been widely used for improving simulation. The computational cost of this method scales with the size of the ensemble, but excessively reducing the ensemble size comes at the cost of reduced predictive performance. The novel procedure proposed in this study is the Entropy Ensemble Filter (EEF), which uses the most informative training data sets in the ensemble rather than all ensemble members created by the bagging method. The results of this study indicate efficiency of the proposed method in application to synthetic data simulation on a sinusoidal signal, a sawtooth signal, and a composite signal. The EEF method can reduce the computational time of simulation by around 50% on average while maintaining predictive performance at the same level of the conventional method, where all of the ensemble models are used for simulation. The analysis of the error gradient (root mean square error of ensemble averages) shows that using the 40% most informative ensemble members of the set initially defined by the user appears to be most effective.

Abstract:Recently, the Entropy Ensemble Filter (EEF) method was proposed to mitigate the computational cost of the Bootstrap AGGregatING (bagging) method. This method uses the most informative training data sets in the model ensemble rather than all ensemble members created by the conventional bagging. In this study, we evaluate, for the first time, the application of the EEF method in Neural Network (NN) modeling of El Nino-southern oscillation. Specifically, we forecast the first five principal components (PCs) of sea surface temperature monthly anomaly fields over tropical Pacific, at different lead times (from 3 to 15 months, with a three-month increment) for the period 1979-2017. We apply the EEF method in a multiple-linear regression (MLR) model and two NN models, one using Bayesian regularization and one Levenberg-Marquardt algorithm for training, and evaluate their performance and computational efficiency relative to the same models with conventional bagging. All models perform equally well at the lead time of 3 and 6 months, while at higher lead times, the MLR model's skill deteriorates faster than the nonlinear models. The neural network models with both bagging methods produce equally successful forecasts with the same computational efficiency. It remains to be shown whether this finding is sensitive to the dataset size.

Abstract. This paper concerns the problem of optimal monitoring network layout using information-theoretical methods. Numerous different objectives based on information measures have been proposed in recent literature, often focusing simultaneously on maximum information and minimum dependence between the chosen locations for data collection stations. We discuss these objective functions and conclude that a single-objective optimization of joint entropy suffices to maximize the collection of information for a given number of stations. We argue that the widespread notion of minimizing redundancy, or dependence between monitored signals, as a secondary objective is not desirable and has no intrinsic justification. The negative effect of redundancy on total collected information is already accounted for in joint entropy, which measures total information net of any redundancies. In fact, for two networks of equal joint entropy, the one with a higher amount of redundant information should be preferred for reasons of robustness against failure. In attaining the maximum joint entropy objective, we investigate exhaustive optimization, a more computationally tractable greedy approach that adds one station at a time, and we introduce the “greedy drop” approach, where the full set of stations is reduced one at a time. We show that no greedy approach can exist that is guaranteed to reach the global optimum.

<p>Probabilistic forecasts are essential for good decision making, because they communicate the forecaster's best attempt at representation of both information available and the remaining uncertainty of a variable of interest. The amount of information provided, which can be measured in bits using information theory, would then be a natural measure of success for the forecast in a verification exercise. On the other hand, it may seem rational to tune the forecasting system to provide maximum value to users. Somewhat counter-intuitively, there are arguments against tuning for maximum value. When the design of the forecasting system also includes the choice of the sources of information, monitoring network optimization becomes part of the problem to solve.  <br>In this presentation, we give a brief overview of the different roles information theory can have in these different aspects of probabilistic forecasting. These roles range from analysis of predictability, model selection, forecast verification, monitoring network design, and data assimilation by ensemble weighting. Using the same theoretical framework for all these aspects has the advantage that some connections can be made that may eventually lead to a more unified perspective on forecasting. </p>