How Robust are the Estimated Effects of Nonpharmaceutical Interventions against COVID-19?

Abstract

There remains much uncertainty about the relative effectiveness of different nonpharmaceutical interventions (NPIs) against COVID-19 transmission. Several studies attempt to infer NPI effectiveness with cross-country, data-driven modelling, by linking from NPI implementation dates to the observed timeline of cases and deaths in a country. These models make many assumptions. Previous work sometimes tests the sensitivity to variations in explicit epidemiological model parameters, but rarely analyses the sensitivity to the assumptions that are made by the choice the of model structure (structural sensitivity analysis). Such analysis would ensure that the inferences made are consistent under plausible alternative assumptions. Without it, NPI effectiveness estimates cannot be used to guide policy. We investigate four model structures similar to a recent state-of-the-art Bayesian hierarchical model. We find that the models differ considerably in the robustness of their NPI effectiveness estimates to changes in epidemiological parameters and the data. Considering only the models that have good robustness, we find that results and policy-relevant conclusions are remarkably consistent across the structurally different models. We further investigate the common assumptions that the effect of an NPI is independent of the country, the time, and other active NPIs. We mathematically show how to interpret effectiveness estimates when these assumptions are violated. DisclaimerThis preprint uses the same data as our previous work (Brauner et al. [2]) and references our previous results at multiple places. However, we have recently updated our data: The new data was collected with independent double entry and has slightly different NPIs, slightly different NPI definitions, and a longer window of analysis. We already updated Brauner et al. [2] to use the new data, but the present preprint is still in the process of being updated. In the meantime, the results differ slightly between the two preprints.

Inferring the effectiveness of government interventions against COVID-19

“…Multiple sensitivity analyses showed how the results changed when we modified the priors over epidemiological parameters, excluded countries from the dataset, used only deaths or confirmed cases as observations, varied the data preprocessing, and more. Finally, we tested our key assumptions by showing results for several alternative models [structural sensitivity (10)] and examined possible confounding of our estimates by unobserved factors influencing R t . In total, we considered NPI effectiveness under 206 alternative experimental conditions (Fig.…”

“…Furthermore, the data are retrospective and observational, meaning that unobserved factors could confound the results. Also, NPI effectiveness estimates can be highly sensitive to arbitrary modeling decisions, as shown by two recent replication studies (9,10). And finally, largescale public NPI datasets suffer from frequent inconsistencies (11) and missing data (12).…”

“…We have collected a large public dataset on NPI implementation dates that has been validated by independent double entry, and we have extensively validated our effectiveness estimates. This validation of data and model is a crucial but often absent or incomplete element of COVID-19 NPI effectiveness studies (10).…”

Estimation and worldwide monitoring of the effective reproductive number of SARS-CoV-2

“…Prior work on the relative impact of specific non-pharmaceutical interventions on R e has shown conflicting results [8, 26, 27, 14, 28, 29]. These differences can be attributed mostly to different model formulations [14, 30, 31], including differing assumptions on the independence of NPIs [30], differing timescales over which the effect of the NPI was analysed [8, 28], whether the time point of the NPI was assumed fixed or allowed to vary [29], and differing geographical scope. There is a need to address whether the strength of measures and the speed of their implementation resulted in a larger and faster decrease of R e , and specifically whether highly restrictive lockdowns were necessary to achieve R e < 1.…”

Comparing Waves of COVID-19 in the US: Scale of response changes over time

“…Much work has been done both to predict the effect of a variety of Non-pharmaceutical interventions (NPIs) on future transmission and to measure the actual effect of NPIs implemented during the pandemic [3, 4, 5, 6, 2]. However, determining which underlying factors contribute to positive or negative response to subsequent pandemic waves in a particular location has proven challenging, partially due to the haphazard way in which NPIs were implemented and the difficulty of measuring individual compliance with state mandates [7, 8, 9]. Despite these challenges, identifying the key factors associated with positive and negative pandemic response should be both of general interest and useful for targeting of public health initiatives.…”