Impact Evaluation of Coronavirus Disease 2019 Policy: A Guide to Common Design Issues

Abstract

Policy responses to COVID-19, particularly those related to non-pharmaceutical interventions, are unprecedented in scale and scope. However, policy impact evaluations require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate strength of evidence in COVID-19 health policy papers. We (1) introduce the basic suite of policy impact evaluation designs for observational data, including cross-sectional analyses, pre/post, interrupted time-series, and difference-in-differences analysis, (2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19, and (3) provide decision-makers and reviewers a conceptual and graphical guide to identifying these key violations. The overall goal of this paper is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.

Measures for infection prevention and control of SARS-CoV-2 in Belgian schools between December 2020 and June 2021: a prospective cohort study

“…Studies examining the impact of IPC measures in schools are limited (14). Design and analysis issues apply, making it difficult to evaluate the impact of individual or combined measures (15). One study estimated that the combination of reduced class density, transmission mitigation measures (use of masks, desk shields, frequent surface cleaning, outdoor instruction), and early identification of active infections would reduce SARS-CoV-2 prevalence (16).…”

Association of Non-Pharmaceutical Interventions to Reduce the Spread of SARS-CoV-2 With Anxiety and Depressive Symptoms: A Multi-National Study of 43 Countries

“…Our results should not be interpreted as indicating that NPIs are causally associated with anxiety and depressive symptoms. When many NPIs are implemented close together in time, which has been the case during the COVID-19 pandemic, isolating their individual effects is exceptionally difficult [ 52 ]. Further, we cannot rule out the presence of unmeasured confounding at either the country or individual levels.…”

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength

“…The full guidance manuscript contains additional explanation and rationale for all parts of this review and the tool, and is available both in the adapted form as was provided to the reviewers in a supplementary file “CHSPER review guidance refs removed.pdf” and in an updated version in Haber et al, 2020. [17] The full review tool is attached as supplementary file “review tool final.pdf”.…”

“…A thorough justification and explanation of how and why these questions were selected is available in the provided guidance document and in Haber et al, 2020. [17]…”

“…Our primary objective was to evaluate each paper for methodological strength and reporting, based on pre-existing review guidance developed for this purpose. [17] As a secondary objective, we also studied our own process: examining the consistency, ease of use, and clarity of this review guidance.…”