A diagram for evaluating multiple aspects of model performance in simulating vector fields

Abstract

Abstract. Vector quantities, e.g., vector winds, play an extremely important role in climate systems. The energy and water exchanges between different regions are strongly dominated by wind, which in turn shapes the regional climate. Thus, how well climate models can simulate vector fields directly affects model performance in reproducing the nature of a regional climate. This paper devises a new diagram, termed the vector field evaluation (VFE) diagram, which is a generalized Taylor diagram and able to provide a concise evaluation of model performance in simulating vector fields. The diagram can measure how well two vector fields match each other in terms of three statistical variables, i.e., the vector similarity coefficient, root mean square length (RMSL), and root mean square vector difference (RMSVD). Similar to the Taylor diagram, the VFE diagram is especially useful for evaluating climate models. The pattern similarity of two vector fields is measured by a vector similarity coefficient (VSC) that is defined by the arithmetic mean of the inner product of normalized vector pairs. Examples are provided, showing that VSC can identify how close one vector field resembles another. Note that VSC can only describe the pattern similarity, and it does not reflect the systematic difference in the mean vector length between two vector fields. To measure the vector length, RMSL is included in the diagram. The third variable, RMSVD, is used to identify the magnitude of the overall difference between two vector fields. Examples show that the VFE diagram can clearly illustrate the extent to which the overall RMSVD is attributed to the systematic difference in RMSL and how much is due to the poor pattern similarity.

Multivariable integrated evaluation of model performance with the vector field evaluation diagram

“…In the same way as for the vector similarity coefficient (VSC) for two-dimensional vector fields (Xu et al, 2016), the VSC for M-dimensional vector fields can be defined as…”

“…1. As for the case of twodimensional vectors (Xu et al, 2016), the RMSLs, i.e., L A and L B , measure the mean and variance of the lengths of vector fields A and B, respectively (Eqs. A4, A5).…”

“…The first pair of datasets is the Climatic Research Unit (CRU) gridded SAT and precipitation (Harris, et al, 2014). The second pair of datasets is the University of Delaware air temperature and precipitation (Willmott and Matsuura, 2001). The third pair of datasets is composed of the Global Historical Climatology Network (GHCN) temperature (Fan and van den Dool, (Schneider et al, 2014).…”

“…Gleckler et al (2008) introduced a suite of metrics, e.g., decomposed mean square error, and relative error metrics, which were used to characterize the model performance for various applications. Xu et al (2016) devised a vector field evaluation (VFE) diagram, which can be regarded as a generalized Taylor diagram, to evaluate the model performance in simulating vector fields, such as vector winds and temperature gradients. Most metrics, e.g., root mean square error, correlation coefficient, and standard devi-ation, measure the model performance in simulating an individual variable .…”

“…Such a constructed vector field integrates multiple variables and can be assessed using the VFE diagram. The VFE diagram can concisely summarize the degree of correspondence between simulated and observed vector fields in terms of multiple statistics (Xu et al, 2016). Therefore, the VFE diagram can be a powerful tool for the MVIE of model performance.…”