Identification and evaluation of epidemic prediction and forecasting reporting guidelines: A systematic review and a call for action

Pollett, Simon; Johansson, Michael A.; Biggerstaff, Matthew; Morton, Lindsay; Bazaco, Sara L.; Major, David M. Brett; Stewart-Ibarra, Anna M.; Pavlin, Julie A.; Mate, Suzanne; Sippy, Rachel; Hartman, Laurie J.; Reich, Nicholas G.; Berry, Irina Maljkovic; Chretien, Jean Paul; Althouse, Benjamin M.; Myer, Diane; Viboud, Cécile; Rivers, Caitlin

doi:10.1016/j.epidem.2020.100400

Cited by 11 publications

(14 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Forecasting research has been defined as research that “ typically offers quantitative statements about an event , outcome , or trend that has not yet been observed , conditional on data that has been observed ,” whereas prediction research is broader and has been defined as a field, which “ may refer to models that examine the mechanistic drivers of epidemiological characteristics … . as well as studies that estimate epidemiological characteristics with inherent forecasting value , such as R 0 ” [ 17 ]. As the 2 terms may be conflated by studies, we recommend that the study is described as a forecast or prediction research in at least the title or abstract (for example, [ 20 – 22 ]).…”

Section: Resultsmentioning

confidence: 99%

“…Briefly, The EPIFORGE guideline concept was registered at the EQUATOR network, and a steering committee ( n = 6) formed to develop a guideline development protocol. Members from this steering committee had already identified a case study that prompted the need for EPIFORGE and conducted a systematic review to ensure no epidemic forecasting reporting guideline existed [ 17 ]. The EPIFORGE steering committee formulated an initial draft checklist of 20 reporting items during 2 teleconferences.…”

Section: Methodsmentioning

confidence: 99%

“…Indeed, many journals mandate that reporting checklists are completed prior to manuscript submission and publication, which has led to demonstrable improvements in study reporting [ 14 , 15 ]. Although principles for policy-driven communication of models for neglected tropical disease programs have been discussed [ 16 ], a recent systematic review noted no reporting guidelines exist specifically for epidemic forecasting and prediction research [ 17 ]. The need for epidemic forecasting reporting guidelines is underscored by a review of Zika forecasting and prediction research, which noted methodological reproducibility, accessibility, and incorporation of uncertainty in these published predictions varied [ 8 ].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

et al. 2021

Self Cite

View full text Add to dashboard Cite

Background The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research. Methods and findings We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies. Conclusions These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The amount of published research on forecasting has doubled in each of the past two decades 32 . As forecasting assumes a more prominent role in scientific inquiry in many disciplines, new guidelines are needed to ensure that forecasting research follows rigorous standards 33 . Additionally, new technology and data architecture are needed to ensure interoperability and comparability between different forecasting systems 23 .…”

Section: Discussionmentioning

confidence: 99%

The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research

et al. 2021

Self Cite

View full text Add to dashboard Cite

Forecasting has emerged as an important component of informed, data-driven decision-making in a wide array of fields. We introduce a new data model for probabilistic predictions that encompasses a wide range of forecasting settings. This framework clearly defines the constituent parts of a probabilistic forecast and proposes one approach for representing these data elements. The data model is implemented in Zoltar, a new software application that stores forecasts using the data model and provides standardized API access to the data. In one real-time case study, an instance of the Zoltar web application was used to store, provide access to, and evaluate real-time forecast data on the order of 108 rows, provided by over 40 international research teams from academia and industry making forecasts of the COVID-19 outbreak in the US. Tools and data infrastructure for probabilistic forecasts, such as those introduced here, will play an increasingly important role in ensuring that future forecasting research adheres to a strict set of rigorous and reproducible standards.

show abstract

“…Unfortunately, no guidelines have been published to standardize the results of epidemic prediction models. 8 It is critically important that the statistical community advocate for transparency in transmission models. Standards for reporting results from transmission models need to be developed as has been done for clinical trials, observational studies, and systematic reviews.…”

mentioning

confidence: 99%

Commentary on the role of statisticians in pandemics

Brookmeyer

2021

Statistics in Medicine

View full text Add to dashboard Cite

I want to express my appreciation for the thoughtful article by Susan Ellenberg and John Morris that compares and contrasts aspects of HIV/AIDS and COVID-19 and the roles that statisticians are playing in addressing the challenges of these pandemics. 1 Public health emergencies require rapid response. Statistical contributions can profoundly inform public policy in health emergencies but only if the contributions are timely. The speed at which the COVID-19 crisis has unfolded has been staggering. The United States surpassed a cumulative COVID-19 death toll of 400 000 within 11 months of the first reported U.S. COVID case. In contrast, the HIV/AIDS cumulative death toll in the United States reached the 400 000 mark more than 16 years after the first AIDS reported case. Statisticians have risen to the enormous challenges presented by the COVID-19 pandemic and are making critical contributions on many fronts.One area that statisticians are contributing is making sense of the myriad of models that try to forecast where the COVID-19 pandemic is heading. All too often COVID-19 model predictions that have appeared both in journals and the popular press are at great odds with each other. 2 Similarly, in the early days of HIV/AIDS, there were widely varying model predictions. For example, one of the first projections of the HIV/AIDS used the normal density to extrapolate AIDS incidence trends. 3 The extrapolation produced the anomalous prediction of rapidly decreasing AIDS cases despite the fact that all the data showed rising AIDS cases and concluded that the projected total size of the AIDS epidemic in the United States would be about 200 000 cases, a number that subsequently was discredited. The authors of that fallacious report cited William Farr and his work with smallpox in the 1830s for justification. Farr's curves for smallpox epidemics looked like normal curves and the normal density was used only by analogy. But of course, smallpox in the 1800s was neither AIDS in the 1990s nor COVID-19 in 2020. One of the earliest prediction models for COVID-19, also based on empirical extrapolation of curves, forecast that COVID-19 deaths in the United States would drop to near zero by summer 2020, 4 a prediction that has also been discredited. 5 Lessons learned from both the COVID-19 and HIV/AIDS pandemics are that extrapolation of empirical trend curves, regardless how complicated the functional forms, can lead to very misleading results. Extrapolating COVID-19 death curves is especially perilous because of the relatively short time period between infection and death, and as such, death counts can take sharp abrupt turns just weeks after occurrence of events that are somewhat unpredictable such as community lockdowns, super-spreader events, and high travel days.One parallel between COVID-19 and HIV/AIDS is that case surveillance data is only measuring the tip of the iceberg of the infected population. Statistical approaches are needed to estimate the size of the "hidden population" which refers to asymptomatic infected perso...

show abstract

Identification and evaluation of epidemic prediction and forecasting reporting guidelines: A systematic review and a call for action

Cited by 11 publications

References 32 publications

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research

Commentary on the role of statisticians in pandemics

Contact Info

Product

Resources

About