Evaluation of Systems-Oriented Public Health Interventions: Alternative Research Designs

Abstract

Keywordscluster randomized controlled trial, stepped wedge design, interrupted time series, multiple baseline design, controlled prepost design AbstractThe need to provide sound evidence of the costs and benefits of real-world public health interventions has driven advances in the development and analysis of designs other than the controlled trial in which individuals are randomized to an experimental condition. Attention to methodological quality is of critical importance to ensure that any evaluation can accurately answer three fundamental questions: (a) Has a change occurred, (b) did the change occur as a result of the intervention, and (c) is the degree of change significant? A range of alternatives to the individual randomized controlled trial (RCT) can be used for evaluating such interventions, including the cluster RCT, stepped wedge design, interrupted time series, multiple baseline, and controlled prepost designs. The key features and complexities associated with each of these designs are explored.

A guide to scaling up population health interventions

“…11,12 In addition, natural experimental studies may be the only option when it is impossible to manipulate exposure to the intervention. 13 As with other quasi-experimental designs, natural experiments cannot definitively determine causation and are subject to bias; however, they can provide a useful inferential tool that may not be obtainable in any other way.…”

Designing and Undertaking a Health Economics Study of Digital Health Interventions

“…Aside from cluster randomization, other study designs such as natural experiments are possible. 17 For example, the five test bed sites within the National Health Service (NHS) England provide a vehicle to examine effectiveness and cost effectiveness on a large scale. 18 However, use of quasi-experimental or observational study designs to demonstrate effectiveness also carries limitations, such as inability to control for unobserved variables.…”

“…18 However, use of quasi-experimental or observational study designs to demonstrate effectiveness also carries limitations, such as inability to control for unobserved variables. 17 More fundamentally, in many cases, an evaluation will be needed by decision makers before the DHI has been trialed, and in cases where a trial does proceed, by the time it is nearing completion, both its effectiveness and cost effectiveness will already be "known" with sufficient accuracy before real-world data are available. This may then provide disincentives for the future use of real-world data to examine effectiveness and cost effectiveness.…”

Evaluation of occupational health interventions using a randomized controlled trial: challenges and alternative research designs

“…Additionally, the stepped wedge design allows for improvement of the intervention based on Schelvis et al lessons learned in every subsequent step (which makes it very suitable for effectiveness trials in practice) and thereby eliminates challenge 4 (the organization or the researcher wants to adjust the intervention protocol). Due to the within and between cluster comparisons at each measurement time across all time periods, this design allows for a variety of conclusions: both shortand long-term effects, fade out effects, and the natural course of the condition under study (26).…”