A prospective evaluation of AI-augmented epidemiology to forecast COVID-19 in the USA and Japan

Abstract

The COVID-19 pandemic has highlighted the global need for reliable models of disease spread. We propose an AI-augmented forecast modeling framework that provides daily predictions of the expected number of confirmed COVID-19 deaths, cases, and hospitalizations during the following 4 weeks. We present an international, prospective evaluation of our models’ performance across all states and counties in the USA and prefectures in Japan. Nationally, incident mean absolute percentage error (MAPE) for predicting COVID-19 associated deaths during prospective deployment remained consistently <8% (US) and <29% (Japan), while cumulative MAPE remained <2% (US) and <10% (Japan). We show that our models perform well even during periods of considerable change in population behavior, and are robust to demographic differences across different geographic locations. We further demonstrate that our framework provides meaningful explanatory insights with the models accurately adapting to local and national policy interventions. Our framework enables counterfactual simulations, which indicate continuing Non-Pharmaceutical Interventions alongside vaccinations is essential for faster recovery from the pandemic, delaying the application of interventions has a detrimental effect, and allow exploration of the consequences of different vaccination strategies. The COVID-19 pandemic remains a global emergency. In the face of substantial challenges ahead, the approach presented here has the potential to inform critical decisions.

Algorithmic fairness in pandemic forecasting: lessons from COVID-19

“…For example, in the Google-Harvard COVID-19 Public Forecasting Model, counties were divided into quartiles across a range of demographic variables, specifically age, gender, median income, race, and population density. The analysis then verified that the model had comparable errors across these groups, using mean absolute percentage error for comparisons 31 . We chose to conduct these fairness analyses at the county level, since distributions of demographic subpopulations tend to be fairly similar at the state level but show substantial variation at the county level.…”

“…When building our COVID-19 model, we chose to include not only mortality, a relatively reliable and hard-to-miss variable, but also additional outputs for cases, hospitalizations, and ICU admissions. Details of the statistical methods underlying the Google-Harvard COVID-19 forecasting model can be found in a separate methods paper 31 . While each output offers value from a policy-planning standpoint, they differ substantially in their reliability.…”

“…In the Google-Harvard forecasting model, we chose to avoid gender- and race-related variables as predictors in our forecasting model, though included age and population density as these were well understood to be directly linked to COVID-19 spread and severity. A more detailed explanation of how socioeconomic factors were incorporated into model transitions is available in the Google-Harvard Public Forecasting Model methods paper 31 . It is important, though, to use socioeconomic variables such as race and gender to stratify model outputs for fairness analyses, even if they are excluded as predictors during model training.…”

Analyzing Patient Trajectories With Artificial Intelligence

“…Such domain shifts might affect the performance of AI solutions [ 92 ], especially when the patient trajectories span a long time horizon. For example, AI-based predictions have been recently applied to patients with COVID-19 to compare the predicted health trajectory with the observed trajectory in a prospective study, finding that the performance of some risk scores decreased over time [ 93 ]. One reason was because of temporal domain shifts over time [ 94 ] as medical professionals learned about the emerging infectious disease and adapted their clinical routines over time, thus yielding different and, in particular, better outcomes than in the data used for training.…”

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

“…This is a prerequisite for model consolidation and improvement, and a need repeatedly expressed. 8 It has been highlighted that such modelling studies should be prospective 9 and ideally follow pre-registered protocols 10 in order to prevent selective reporting and hindsight bias (i.e., the tendency to overstate the predictability of past events in hindsight).…”