Evaluating Forecasts with scoringutils in R

Bosse, Nikos I.; Gruson, Hugo; Cori, Anne; Leeuwen, Edwin van; Funk, Sebastian; Abbott, Sam

doi:10.48550/arxiv.2205.07090

Cited by 8 publications

(4 citation statements)

References 26 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the overall skill for nine of the 10 models was similar, regression analyses identified specific differences in predicted skill based on historical case counts and observed case counts that provide insight on forecast failures. For all predictions, we found a general association between higher observed values and increased surprisal (worse skill) as has been noted in other forecasting studies (Bosse et al, 2022). Accounting for this relationship, we found important between-model differences in skill for different scenarios.…”

Section: Discussionsupporting

confidence: 85%

Multi‐Model Prediction of West Nile Virus Neuroinvasive Disease With Machine Learning for Identification of Important Regional Climatic Drivers

Holcomb,

Staples,

Nett

et al. 2023

GeoHealth

View full text Add to dashboard Cite

West Nile virus (WNV) is the leading cause of mosquito‐borne illness in the continental United States (CONUS). Spatial heterogeneity in historical incidence, environmental factors, and complex ecology make prediction of spatiotemporal variation in WNV transmission challenging. Machine learning provides promising tools for identification of important variables in such situations. To predict annual WNV neuroinvasive disease (WNND) cases in CONUS (2015–2021), we fitted 10 probabilistic models with variation in complexity from naïve to machine learning algorithm and an ensemble. We made predictions in each of nine climate regions on a hexagonal grid and evaluated each model's predictive accuracy. Using the machine learning models (random forest and neural network), we identified the relative importance and variation in ranking of predictors (historical WNND cases, climate anomalies, human demographics, and land use) across regions. We found that historical WNND cases and population density were among the most important factors while anomalies in temperature and precipitation often had relatively low importance. While the relative performance of each model varied across climatic regions, the magnitude of difference between models was small. All models except the naïve model had non‐significant differences in performance relative to the baseline model (negative binomial model fit per hexagon). No model, including the ensemble or more complex machine learning models, outperformed models based on historical case counts on the hexagon or region level; these models are good forecasting benchmarks. Further work is needed to assess if predictive capacity can be improved beyond that of these historical baselines.

show abstract

Section: Discussionsupporting

confidence: 85%

Multi‐Model Prediction of West Nile Virus Neuroinvasive Disease With Machine Learning for Identification of Important Regional Climatic Drivers

Holcomb,

Staples,

Nett

et al. 2023

GeoHealth

View full text Add to dashboard Cite

show abstract

“…For a practical comparison, we take advantage of the fact that a wide variety of forecasts are submitted to the European COVID-19 Forecast Hub [9] and to the COVID-19 Forecast Hub [10]. A study on the methodology to evaluate and compare forecast has been proposed in [11], using the data of this Hub. We shall address the theoretical comparison in section 3.…”

Section: Resultsmentioning

confidence: 99%

Learning from the past: a short term forecast method for the COVID-19 incidence curve

Morel

Álvarez

2022

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemy has created a radically new situation where most countries provide raw measurements of their daily incidence and disclose them in real time. This enables new machine learning forecast strategies where the prediction might no longer be based just on the past values of the current incidence curve, but could take advantage of observations in many countries. We present such a simple global machine learning procedure using all past daily incidence trend curves. Each of the 27,418 COVID-19 incidence trend curves in our database contains the values of 56 consecutive days extracted from observed incidence curves across 61 word regions and countries. Given a current incidence trend curve observed over the past four weeks, its forecast in the next four weeks is computed by matching it with the first four weeks of all samples, and ranking them by their similarity to the query curve. Then the 28 days forecast is obtained by a statistical estimation combining the values of the 28 last observed days in those similar samples. Using comparison performed by the European Covid-19 Forecast Hub with the current state of the art forecast methods, we verify that the proposed global learning method, EpiLearn, compares favorably to methods forecasting from a single past curve.

show abstract

“…The WIS is a proper scoring rule that generalises the absolute error and gives penalties for interval spread as well as for over- and underprediction [27]. All three metrics (AE, ECR, WIS) were calculated using the scoringutils package [28]. We used the default summary function implemented in scoringutils (i.e.…”

Section: Methodsmentioning

confidence: 99%

Real-time forecasting of COVID-19-related hospital strain in France using a non-Markovian mechanistic model

Massey

Boennec

Restrepo‐Ortiz

et al. 2023

Preprint

View full text Add to dashboard Cite

Background The COVID-19 pandemic emphasised the importance of access to reliable real-time forecasts for key epidemiological indicators. Given the strong heterogeneity between regions, providing forecasts at the local level is essential for health professionals. Methods We developed a SARS-CoV-2 transmission model in France, COVIDici, that performs parameter estimation using up-to-date vaccination coverage and hospital data to provide forecasts up to a four-week horizon based on the current epidemic trend. We present the model, its associated online tool and perform a retrospective evaluation of the forecasts provided from January to December 2021 by comparing to three standard statistical forecasting methods (auto-regression, exponential smoothing, and ARIMA) at the national and regional levels. Results COVIDici allowed simultaneous real-time visualisation of several indicators of the COVID-19 epidemic at the sub-national level. For anticipating risk of critical care unit overload, it performed worse compared to the baseline methods for forecasts under the three-week horizon, but had better point forecasts at the longest horizons (e.g. four weeks) for 8 of the 13 regions considered depending on the metric. Conclusions Effective communication between modelers and clinicians is essential for utilising forecasts for health care planning. Online visualisation tools and consideration of how metrics can be affected by distortion from non-pharmaceutical government interventions facilitate this dialogue.

show abstract

Evaluating Forecasts with scoringutils in R

Cited by 8 publications

References 26 publications

Multi‐Model Prediction of West Nile Virus Neuroinvasive Disease With Machine Learning for Identification of Important Regional Climatic Drivers

Multi‐Model Prediction of West Nile Virus Neuroinvasive Disease With Machine Learning for Identification of Important Regional Climatic Drivers

Learning from the past: a short term forecast method for the COVID-19 incidence curve

Real-time forecasting of COVID-19-related hospital strain in France using a non-Markovian mechanistic model

Contact Info

Product

Resources

About