Toward a true spatial model evaluation in distributed hydrological modeling: Kappa statistics, Fuzzy theory, and EOF‐analysis benchmarked by the human perception and evaluated against a modeling case study

Abstract

The hydrological modeling community is aware that the validation of distributed hydrological models has to move beyond aggregated performance measures, like hydrograph assessment by means of Nash-Suitcliffe efficiency toward a true spatial model validation. Remote sensing facilitates continuous data and can be measured on a similar spatial scale as the predictive scale of the hydrological model thereby it can serve as suitable data for the spatial validation. The human perception is often described as a very reliable and well-trained source for pattern comparison, which this study wants to exploit. A web-based survey that is interpreted based on approximately 200 replies reflects the consensus of the human perception on map comparisons of a reference map and 12 synthetic perturbations. The resulting similarity ranking can be used as a reference to benchmark various spatial performance metrics. This study promotes Fuzzy theory as a suitable approach because it considers uncertainties related to both location and value in the simulated map. Additionally, an EOF-analysis (Empirical Orthogonal Function) is conducted to decompose the map comparison into its similarities and dissimilarities. A modeling case study serves to further examine the metrics capability to assess the goodness of fit between simulated and observed land surface temperature maps. The EOF-analysis unambiguously identifies a systematic depth to groundwater table-related model deficiency. Kappa statistic extended by Fuzziness is a suitable and commonly applied measure for map comparison. However, its apparent bias sensitivity limits it's capability as a diagnostic tool to detect the distinct deficiency.

Combining satellite data and appropriate objective functions for improved spatial pattern performance of a distributed hydrologic model

“…Therefore, more flexible yet physically meaningful parameterisations were implemented where full spatial variability was enabled by combining 2-3 calibration parameters to initial spatial distributions of soil type and LAI. Regarding the use of appropriate spatial performance metrics, the initial attempts using standard metrics of correlation coefficient, Mapcurves (Hargrove et al, 2006), coef- ficient of variation, Goodman and Kruskal's lambda (Goodman and Kruskal, 1954), agreement coefficient (Ji and Gallo, 2006), Theil's uncertainty, EOF, and Cramér's V (Cramér, 1946;Koch et al, 2015;Rees, 2008) proved to be inadequate in a calibration framework, since undesired visual patterns were achieved, e.g. with high correlation, but toolow standard deviation or highly separate clusters.…”

“…In this study, we compare AET from TSEB (in W m −2 ) based on instantaneous satellite data with daily averaged AET (mm day −1 ) simulated by the model and regard the satellite-based AET maps as the "observation" even though they are more accurately AET "estimates" based on satellite observations. Attempts to use numerous other spatial metrics including Mapcurves, fractions skill score (FSS), Goodman and Kruskal's lambda, Theil's Uncertainty, empirical orthogonal functions (EOFs) and Cramér's V (Cramér, 1946;Koch et al, 2015;Rees, 2008) did not distinguish the general AET patterns or the spatial efficiency metric. The strength of the spatial efficiency metric is that each component contains different and non-overlapping information.…”

Calibration of a large-scale hydrological model using satellite-based soil moisture and evapotranspiration products

“…Furthermore, the validation of this study was carried out exclusively on streamflow. Other validation approaches, including the empirical orthogonal functions, wavelet analysis or their combination, may be another promising way towards a more in-depth validation of distributed hydrological models (Mascaro et al, 2015;Koch et al, 2015;Fang et al, 2015).…”

Hydrological Modelling using Satellite-Based Crop Coefficients: A Comparison of Methods at the Basin Scale

“…Calibration performance coefficients-for example those used in this work-could be used to assess which method performs better than others. A more in-depth analysis could be performed for this evaluation as well, by using spatial metrics that assess the degree of similarity between of the spatial patterns in the model simulations [66,67].…”