Evaluating epidemic forecasts in an interval format

Bracher, Johannes; Ray, Evan L.; Gneiting, Tilmann; Reich, Nicholas G.

doi:10.1371/journal.pcbi.1008618

Cited by 172 publications

(229 citation statements)

References 32 publications

Supporting

Mentioning

206

Contrasting

Order By: Relevance

“…CRPS is therefore useful in this study as it serves as a measure of accuracy that can be applied to a consensus distribution constructed from participants’ 75% confidence intervals, rather than from their point estimates. An alternative, perhaps more straightforward approach would have been to use the weighted interval score [ 22 ], which approximates the CRPS and does not require the construction of a full predictive distribution. Constructing consensus distributions also allowed us to better visualize the aggregated predictions.…”

Section: Methodsmentioning

confidence: 99%

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

2021

View full text Add to dashboard Cite

Throughout the COVID-19 pandemic, social and traditional media have disseminated predictions from experts and nonexperts about its expected magnitude. How accurate were the predictions of ‘experts’—individuals holding occupations or roles in subject-relevant fields, such as epidemiologists and statisticians—compared with those of the public? We conducted a survey in April 2020 of 140 UK experts and 2,086 UK laypersons; all were asked to make four quantitative predictions about the impact of COVID-19 by 31 Dec 2020. In addition to soliciting point estimates, we asked participants for lower and higher bounds of a range that they felt had a 75% chance of containing the true answer. Experts exhibited greater accuracy and calibration than laypersons, even when restricting the comparison to a subset of laypersons who scored in the top quartile on a numeracy test. Even so, experts substantially underestimated the ultimate extent of the pandemic, and the mean number of predictions for which the expert intervals contained the actual outcome was only 1.8 (out of 4), suggesting that experts should consider broadening the range of scenarios they consider plausible. Predictions of the public were even more inaccurate and poorly calibrated, suggesting that an important role remains for expert predictions as long as experts acknowledge their uncertainty.

show abstract

Section: Methodsmentioning

confidence: 99%

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

2021

View full text Add to dashboard Cite

show abstract

“…The score has the intuitive interpretation that it rewards narrow intervals, with observations that fall outside the interval incurring a penalty, the magnitude of which depends on the value of α ( Gneiting and Raftery, 2007 ). In an application to influenza forecasting, Bracher et al (2021) use this interpretation to seek insight into why the interval score for one forecasting model is lower than another.…”

Section: Combining Interval Forecasts Of Covid-19 Mortalitymentioning

confidence: 99%

“…This is a proper score, and this can be seen by viewing it as a quantile-weighted version of the CRPS (see Gneiting and Ranjan, 2011 ). We note that Bracher et al (2021) present it as a weighted sum of the interval score of expression (2) and the quantile score of expression (1) for the median.…”

Section: Combining Distributional Forecasts Of Covid-19 Mortalitymentioning

confidence: 99%

“…In this paper we use the most common, which is that a (1- α ) interval forecast is an interval predicted to contain the true outcome with probability (1- α ), and equal probability of being above or below the interval. Interval forecasts are valuable, as they can support real-time decision making and provide situational awareness (see, for example, Grushka-Cockayne and Jose, 2020 ; Bracher et al, 2021 ). In this paper, we consider the case where probabilistic forecasts of COVID-19 deaths are provided by multiple forecasters.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Combining probabilistic forecasts of COVID-19 mortality in the United States

Taylor

2023

European Journal of Operational Research

View full text Add to dashboard Cite

The COVID-19 pandemic has placed forecasting models at the forefront of health policy making. Predictions of mortality, cases and hospitalisations help governments meet planning and resource allocation challenges. In this paper, we consider the weekly forecasting of the cumulative mortality due to COVID-19 at the national and state level in the U.S. Optimal decision-making requires a forecast of a probability distribution, rather than just a single point forecast. Interval forecasts are also important, as they can support decision making and provide situational awareness. We consider the case where probabilistic forecasts have been provided by multiple forecasting teams, and we combine the forecasts to extract the wisdom of the crowd. We use a dataset that has been made publicly available from the COVID-19 Forecast Hub. A notable feature of the dataset is that the availability of forecasts from participating teams varies greatly across the 40 weeks in our study. We evaluate the accuracy of combining methods that have been previously proposed for interval forecasts and predictions of probability distributions. These include the use of the simple average, the median, and trimming methods. In addition, we propose several new weighted combining methods. Our results show that, although the median was very useful for the early weeks of the pandemic, the simple average was preferable thereafter, and that, as a history of forecast accuracy accumulates, the best results can be produced by a weighted combining method that uses weights that are inversely proportional to the historical accuracy of the individual forecasting teams.

show abstract

“…Evaluating probabilistic predictions requires different scoring functions than point estimates do. For a comprehensive discussion of the topic, we refer the interested reader to [18]. Here, we briefly summarize the main evaluation measures used in this work.…”

Section: Evaluating Nowcastsmentioning

confidence: 99%

CorCast: A Distributed Architecture for Bayesian Epidemic Nowcasting and its Application to District-Level SARS-CoV-2 Infection Numbers in Germany

Hildebrandt

Bob

Teschner

et al. 2021

Preprint

View full text Add to dashboard Cite

Timely information on current infection numbers during an epidemic is of crucial importance for decision makers in politics, medicine, and businesses. As information about local infection risk can guide public policy as well as individual behavior, such as the wearing of personal protective equipment or voluntary social distancing, statistical models providing such insights should be transparent and reproducible as well as accurate. Fulfilling these requirements is drastically complicated by the large amounts of data generated during exponential growth of infection numbers, and by the complexity of common inference pipelines. Here, we present CorCast -- a stable and scalable distributed architecture for the reproducible estimation of nowcasts suitable for pandemic scenarios -- and its application to the inference of district-level SARS-CoV-2 infection numbers in Germany.

show abstract

Evaluating epidemic forecasts in an interval format

Cited by 172 publications

References 32 publications

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

Combining probabilistic forecasts of COVID-19 mortality in the United States

CorCast: A Distributed Architecture for Bayesian Epidemic Nowcasting and its Application to District-Level SARS-CoV-2 Infection Numbers in Germany

Contact Info

Product

Resources

About