Conceptualising natural and quasi experiments in public health

Abstract

Background Natural or quasi experiments are appealing for public health research because they enable the evaluation of events or interventions that are difficult or impossible to manipulate experimentally, such as many policy and health system reforms. However, there remains ambiguity in the literature about their definition and how they differ from randomized controlled experiments and from other observational designs. We conceptualise natural experiments in the context of public health evaluations and align the study design to the Target Trial Framework. Methods A literature search was conducted, and key methodological papers were used to develop this work. Peer-reviewed papers were supplemented by grey literature. Results Natural experiment studies (NES) combine features of experiments and non-experiments. They differ from planned experiments, such as randomized controlled trials, in that exposure allocation is not controlled by researchers. They differ from other observational designs in that they evaluate the impact of events or process that leads to differences in exposure. As a result they are, in theory, less susceptible to bias than other observational study designs. Importantly, causal inference relies heavily on the assumption that exposure allocation can be considered ‘as-if randomized’. The target trial framework provides a systematic basis for evaluating this assumption and the other design elements that underpin the causal claims that can be made from NES. Conclusions NES should be considered a type of study design rather than a set of tools for analyses of non-randomized interventions. Alignment of NES to the Target Trial framework will clarify the strength of evidence underpinning claims about the effectiveness of public health interventions.

Preterm births in South-West England before and during the COVID-19 pandemic: an audit of retrospective data

“…We recognise that the limitation of our observations is reliant on a crude analysis of retrospective data and does not include possible explanatory factors, including induction of labour and/or other obstetric complications or any socioeconomic factors. Although a time-series analysis or alternative quasi-experimental approach could potentially provide a more robust analysis [ 15 ], the utility of such analysis here is limited by the lack of data around specific date cut-points, the unknown timing of potential impact of the various COVID-related lockdowns/restrictions, and relatively few post-lockdown time-points. In addition to lack of weekly data, stillbirths and individual level data were also not available further limiting such an approach.…”

A nationwide school fruit and vegetable policy and childhood and adolescent overweight: A quasi-natural experimental study

“…From 2007 to 2014, the Norwegian government implemented a nationwide school FFV provision policy for lower secondary schools (pupils age 13–15 years). Since approximately one-third of elementary schools are combined with lower secondary schools, elementary age children (6–12 years) attending a combined school also received FFVs while those attending a pure elementary school did not receive FFVs, providing a nationwide quasi-natural experimental setting for policy evaluation [ 20 ]. Our objective was to assess whether exposure to the nationwide FFV policy for up to 4 years from starting school resulted in any benefits or unintended consequences with respect to childhood and early adolescent BMI and weight status.…”

Changes in household food and drink purchases following restrictions on the advertisement of high fat, salt, and sugar products across the Transport for London network: A controlled interrupted time series analysis

“…We used a CITS study design as a robust approach to evaluate a natural experiment where a randomised design was not feasible or pragmatic [ 35 ]. Such studies are conducted in real-world settings [ 35 , 40 ], and can provide evidence to inform policy [ 46 ]. Use of a control group reduced the risk of national-level, time-varying confounders driving observed results, such as seasonal effects, underlying trends in HFSS purchasing, and the effect of other sugar and calorie reduction strategies such as the SDIL [ 47 – 49 ].…”

“…Follow-up was conservatively reduced to 10 months to avoid contamination of outcomes as a result of the early stages of the COVID-19 pandemic (reductions in the use of public transport and the early stages of panic buying). We conducted sensitivity analyses (described above) that were not pre-specified in our protocol, to assess the robustness of our results [ 40 ].…”