Deterministic and fuzzy verification methods for a hierarchy of numerical models

Abstract

Deterministic and fuzzy verification methods are compared to assess the Quantitative Precipitation Forecasts (QPF) performance of an hierarchy of models run at Météo-France: two operational models (ARPEGE and ALADIN) and a prototype version of the high resolution model AROME. The reference data are 24-h accumulated rainfall values measured by the French climatological network of rain gauges. The deterministic forecasts are converted to frequencies of threshold exceedence in a neighbourhood in order to apply a fuzzy verification method, able to determine the influence of the double penalty. Local and regional versions of the Brier skill score (BSS) are computed and persistence is the selected reference system. An optimal size of the neighbourhood can be determined from the local version of the BSS. The regional version of the BSS increases with the size of the neighbourhood and provides useful information on the reduction of the error for large scales. All the scores show that the ALADIN 3DVAR implementation improves the quality of the ALADIN QPF in comparison to the ARPEGE QPF. Deterministic verification methods show that AROME improves the QPF for weak precipitation but produces too many false alarms for heavy rain. Moreover, the fuzzy approach removes more substantially the double penalty for the heavy rain forecasts of AROME than for the light precipitation of ALADIN. After recalibration the ALADIN QPF have better deterministic scores than the AROME QPF. Nevertheless, fuzzy verification methods prove the contrary: for scales larger than 50 km, the AROME scores exceed the ALADIN scores, but not significantly.

Verification of the French operational high‐resolution model AROME with the regional Brier probability score

“…The BSS_NO increases with the size of the neighbourhood because it tolerates increasingly large errors in positioning events. The BSS_NO tends towards the root-mean-square of the daily bias (Amodei and Stein, 2009). The smaller neighbourhood corresponds to an AROME grid point and one can show that it is thus equal to the HSS already plotted in Figure 3(c).…”

“…At Météo-France, Amodei and Stein (2009) used the regional Brier skill score (BSS_NO) to control the rainfall simulated by a hierarchy of operational models.…”

Generalization and application of the flux-conservative thermodynamic equations in the AROME model of the ALADIN system

“…Figures 4 and 5 compare the forecasted precipitation over the two periods. To avoid the problem of the double penalty, the precipitation is verified with the neighbourhood observation Brier skill score (Amodei and Stein, 2009). This score is determined by calculating the probability that a precipitation threshold is exceeded in the vicinity of an observation.…”

La vérification des prévisions météorologiques à Météo-France

“…Des scores alternatifs ont été proposés afin de limiter l'impact de cette double peine : on trouve des propositions originales dans Roberts et Lean (2008) et une synthèse dans Eberth (2008). Un tel score alternatif pour évaluer les modèles HR est opé-rationnel à Météo-France (Amodei et Stein, 2009). Le principe est de remplacer la vérif ication en un point donné par la vérif ication sur un domaine ou voisinage comprenant un nombre important de points de grille.…”