Republished: How to study improvement interventions: a brief overview of possible study types

Abstract

Improvement (defined broadly as purposive efforts to secure positive change) has become an increasingly important activity and field of inquiry within healthcare. This article offers an overview of possible methods for the study of improvement interventions. The choice of available designs is wide, but debates continue about how far improvement efforts can be simultaneously practical (aimed at producing change) and scientific (aimed at producing new knowledge), and whether the distinction between the practical and the scientific is a real and useful one. Quality improvement projects tend to be applied and, in some senses, self-evaluating. They are not necessarily directed at generating new knowledge, but reports of such projects if well conducted and cautious in their inferences may be of considerable value. They can be distinguished heuristically from research studies, which are motivated by and set out explicitly to test a hypothesis, or otherwise generate new knowledge, and from formal evaluations of improvement projects. We discuss variants of trial designs, quasi-experimental designs, systematic reviews, programme evaluations, process evaluations, qualitative studies, and economic evaluations. We note that designs that are better suited to the evaluation of clearly defined and static interventions may be adopted without giving sufficient attention to the challenges associated with the dynamic nature of improvement interventions and their interactions with contextual factors. Reconciling pragmatism and research rigour is highly desirable in the study of improvement. Trade-offs need to be made wisely, taking into account the objectives involved and inferences to be made.

A controlled trial of implementing a complex mental health intervention for carers of vulnerable young people living in out-of-home care: the ripple project

“…Although the absence of longitudinal data is a limitation of the study, the current design is a useful way to evaluate a complex intervention [63]. We anticipate that future studies will build on this foundation to develop cohort studies (to identify modifiable factors important to the mental health of young women and men of different ages) and more complex designs such as stepped-wedge cluster RCTs (to test the implementation and adaptation of mental health interventions in various types of OOHC and for specific groups of young people).…”

“…While there are challenges associated with this trial design, comparing local regions is the most pragmatic test of effectiveness in a real-world context, particularly if long-term or external influences on the settings are accounted for. [63, 64] A preliminary test of the comparability of the populations and settings is reported in the census data below.

Fig. 2Design of the Ripple controlled trial of implementing a complex mental health intervention in out-of-home care (OoHC) settings in Melbourne

…”

A framework for learning about improvement: embedded implementation and evaluation design to optimize learning

“…Embedded evaluation design requires evaluators to interact with improvers at the design level of the improvement initiative, as well as with improvers who are on the ground coaching teams and facilities in implementing the improvement. There is no single ‘correct’ model; the design should be fit-for-purpose and reflect the expectations of the audience for the evaluation [ 7 – 9 ]. For example, a governmental agency may demand a model that maximizes objectivity and assesses quantitatively whether the intervention ‘worked’.…”

How do we learn about improving health care: a call for a new epistemological paradigm

“…A 2015 literature review by Portela et al . [ 15 ] provided a useful overview of various methods that can be used to learn about improving health care and describes the strengths and weakness of various approaches that range from more traditional, experimental designs to quasi-experimental designs, as well as systematic reviews, program and process evaluations, qualitative methods and economic evaluations. The authors note that the dichotomy between designs classified as practical (‘aimed at producing change’) and those classified as scientific (‘aimed at producing new knowledge’) may be a false one, and that the field should find ways to optimize rigor and generalizability of studies, without compromising the importance of adaptability and context.…”