Sample Stratification in Verification of Ensemble Forecasts of Continuous Scalar Variables: Potential Benefits and Pitfalls

Bellier, Joseph; Zin, Isabella; Bontron, Guillaume

doi:10.1175/mwr-d-16-0487.1

Cited by 30 publications

(29 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Forecast stratification (Broecker, 2008), as the process of dividing the whole dataset into different subsets and computing verification metrics for each subset, has been introduced as a way to better diagnose where the deficiencies of the forecast system lie. Bellier et al (2017) compared different strategies for the stratification criteria, based on either the observations or the forecasts, and justified the use of a forecast-based stratification criteria for verification rank histograms. Indeed, they showed that conditionning the rank histogram to observations is likely to draw 20 erroneous conclusions about the real behaviour of ensemble forecasts.…”

mentioning

confidence: 99%

Statistical post-processing of ensemble forecasts of the height of new snow

Nousu¹,

Lafaysse²,

Vernay³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Forecasting the height of new snow (HN) is crucial for avalanche hazard forecasting, roads viability, ski resorts management and tourism attractiveness. Meteo-France operates the PEARP-S2M probabilistic forecasting system including 35 members of the PEARP Numerical Weather Prediction system, where the SAFRAN downscaling tool is refining the elevation resolution, and the Crocus snowpack model is representing the main physical processes in the snowpack. It provides better HN forecasts 5 than direct NWP diagnostics but exhibits significant biases and underdispersion. We applied a statistical post-processing to these ensemble forecasts, based on Nonhomogeneous Regression with a censored shifted Gamma distribution. Observations come from manual measurements of 24-hour HN in French Alps and Pyrenees. The calibration is tested at the station-scale and the massif-scale (i.e. aggregating different stations over areas of 1000 km 2 ). Compared to the raw forecasts, similar improvements are obtained for both spatial scales. Therefore, the post-processing can be applied at any point of the massifs. Two 10 training datasets are tested: (1) a 22-year homogeneous reforecast for which the NWP model resolution and physical options are identical to the operational system but without the same initial perturbations; (2) 3-year real-time forecasts with a heterogeneous model configuration but the same perturbation methods. The impact of the training dataset depends on lead time and on the evaluation criteria. The long-term reforecast improves the reliability of severe snowfall but leads to overdispersion due to the discrepancy in real-time perturbations. Thus, the development of reliable automatic forecasting products of HN needs 15 long reforecasts as homogeneous as possible with the operational systems.30 2018) are considerably lower than errors in solid precipitation measurements.The goal of this study is to test the ability of a nonhomogeneous regression method to improve the ensemble forecasts of HN from the PEARP-S2M ensemble snowpack modelling system. More precisely, the regression method of Scheuerer andHamill (2015) based on the Censored Shifted Gamma Distribution was chosen in this work for the advantages identified by the authors in the case of precipitation forecasts. In particular, this method allows to extrapolate the statistical relationship 35 3 https://doi.between predictors and predictands from common events to more unusual events. Considering the specificities of the available datasets in terms of predictands and predictors, two other scientific questions are considered: (1) can statistical postprocessing be applied at a larger spatial scale than the observation points? (2) what are the requirements of a robust training forecast dataset for statistical postprocessing?The structure of the paper is as follows. Section 2 describes the model components of the PEARP-S2M system, the obser-5 vation and forecast datasets used in this study, the nonhomogeneous regression method chosen for post-processing and the evaluation me...

show abstract

mentioning

confidence: 99%

Statistical post-processing of ensemble forecasts of the height of new snow

Nousu¹,

Lafaysse²,

Vernay³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…8.) Note that the stratification should be based on the forecasts (e.g., raw forecasts mean as we used here) instead of the observations to ensure the reliability for each stratum (Bellier et al 2017;Lerch et al 2017). More details of the verification metrics are described in the appendix A and related references (Wilks 2011).…”

Section: B Experimental Design and Verification Methodsmentioning

confidence: 99%

Factors Influencing the Performance of Regression-Based Statistical Postprocessing Models for Short-Term Precipitation Forecasts

Duan

Wang

2019

Weather and Forecasting

View full text Add to dashboard Cite

Statistical postprocessing models can be used to correct bias and dispersion errors in raw precipitation forecasts from numerical weather prediction models. In this study, we conducted experiments to investigate four factors that influence the performance of regression-based postprocessing models with normalization transformations for short-term precipitation forecasts. The factors are 1) normalization transformations, 2) incorporation of ensemble spread as a predictor in the model, 3) objective function for parameter inference, and 4) two postprocessing schemes, including distributional regression and joint probability models. The experiments on the first three factors are based on variants of a censored regression model with conditional heteroscedasticity (CRCH). For the fourth factor, we compared CRCH as an example of the distributional regression with a joint probability model. The results show that the CRCH with normal quantile transformation (NQT) or power transformation performs better than the CRCH with log–sinh transformation for most of the subbasins in Huai River basin with a subhumid climate. The incorporation of ensemble spread as a predictor in CRCH models can improve forecast skill in our research region at short lead times. The influence of different objective functions (minimum continuous ranked probability score or maximum likelihood) on postprocessed results is limited to a few relatively dry subbasins in the research region. Both the distributional regression and the joint probability models have their advantages, and they are both able to achieve reliable and skillful forecasts.

show abstract

“…First, the PEARP reforecasts consist of 10 members, including 1 control member, and were issued in 2018. The reforecasts (Boisserie et al, 2016) are based on a homogeneous model configuration identical to the operational release of 5 December 2017 (same resolution and physical parameterizations), but they only include physical perturbations and no perturbation of the initial state, contrary to operational PEARP forecasts. The initial states are built with ERA-Interim reanalysis (Dee et al, 2011) for the atmospheric variables and by the 24 h stand-alone coupled forecasts of the SURFEX/ARPEGE model for the Earth parameters.…”

Section: Reforecasts Used For Trainingmentioning

confidence: 99%

Statistical post-processing of ensemble forecasts of the height of new snow

Nousu

Lafaysse

Vernay

et al. 2019

Nonlin. Processes Geophys.

Self Cite

View full text Add to dashboard Cite

Abstract. Forecasting the height of new snow (HN) is crucial for avalanche hazard forecasting, road viability, ski resort management and tourism attractiveness. Météo-France operates the PEARP-S2M probabilistic forecasting system, including 35 members of the PEARP Numerical Weather Prediction system, where the SAFRAN downscaling tool refines the elevation resolution and the Crocus snowpack model represents the main physical processes in the snowpack. It provides better HN forecasts than direct NWP diagnostics but exhibits significant biases and underdispersion. We applied a statistical post-processing to these ensemble forecasts, based on non-homogeneous regression with a censored shifted Gamma distribution. Observations come from manual measurements of 24 h HN in the French Alps and Pyrenees. The calibration is tested at the station scale and the massif scale (i.e. aggregating different stations over areas of 1000 km2). Compared to the raw forecasts, similar improvements are obtained for both spatial scales. Therefore, the post-processing can be applied at any point of the massifs. Two training datasets are tested: (1) a 22-year homogeneous reforecast for which the NWP model resolution and physical options are identical to the operational system but without the same initial perturbations; (2) 3-year real-time forecasts with a heterogeneous model configuration but the same perturbation methods. The impact of the training dataset depends on lead time and on the evaluation criteria. The long-term reforecast improves the reliability of severe snowfall but leads to overdispersion due to the discrepancy in real-time perturbations. Thus, the development of reliable automatic forecasting products of HN needs long reforecasts as homogeneous as possible with the operational systems.

show abstract

Sample Stratification in Verification of Ensemble Forecasts of Continuous Scalar Variables: Potential Benefits and Pitfalls

Cited by 30 publications

References 41 publications

Statistical post-processing of ensemble forecasts of the height of new snow

Statistical post-processing of ensemble forecasts of the height of new snow

Factors Influencing the Performance of Regression-Based Statistical Postprocessing Models for Short-Term Precipitation Forecasts

Statistical post-processing of ensemble forecasts of the height of new snow

Contact Info

Product

Resources

About