Challenges to evaluating complex interventions: a content analysis of published papers

Abstract

BackgroundThere is continuing interest among practitioners, policymakers and researchers in the evaluation of complex interventions stemming from the need to further develop the evidence base on the effectiveness of healthcare and public health interventions, and an awareness that evaluation becomes more challenging if interventions are complex.We undertook an analysis of published journal articles in order to identify aspects of complexity described by writers, the fields in which complex interventions are being evaluated and the challenges experienced in design, implementation and evaluation. This paper outlines the findings of this documentary analysis.MethodsThe PubMed electronic database was searched for the ten year period, January 2002 to December 2011, using the term “complex intervention*” in the title and/or abstract of a paper. We extracted text from papers to a table and carried out a thematic analysis to identify authors’ descriptions of challenges faced in developing, implementing and evaluating complex interventions.ResultsThe search resulted in a sample of 221 papers of which full text of 216 was obtained and 207 were included in the analysis. The 207 papers broadly cover clinical, public health and methodological topics. Challenges described included the content and standardisation of interventions, the impact of the people involved (staff and patients), the organisational context of implementation, the development of outcome measures, and evaluation.ConclusionsOur analysis of these papers suggests that more detailed reporting of information on outcomes, context and intervention is required for complex interventions. Future revisions to reporting guidelines for both primary and secondary research may need to take aspects of complexity into account to enhance their value to both researchers and users of research.

Lessons from Complex Interventions to Improve Health

“…The Canadian Institutes of Health Research demonstrated their interest by funding the International Collaboration on Complex Interventions (involving investigators from Canada, the United States, the United Kingdom, and Australia), which produced some of the key papers in these debates (7,16,31,34,36,55,62,65,67,94,96,98,110,120,128,129). This collaboration also enabled a partnership with the Centers for Disease Control and Prevention and the Prevention Research Centers with conferences in Chicago in 2009 (128), Montréal in 2010 (27), and Toronto in 2011 to advance theories and methods.…”

Putting context centre stage: evidence from a systems evaluation of an area based empowerment initiative in England

“…By definition, such actions can be broadly conceived of as social interventions, as they are, in effect, purposeful attempts to change social contexts (as well as economic and environmental factors) that may in part influence patterns of lifestyle, behaviours and health outcomes (Poland et al, 2008). Despite its importance, a more nuanced interpretation of context is still not routinely integrated into the evaluation of initiatives in the public health field (Datta & Petticrew, 2013). This is particularly the case in relation to social aspects of context (Shoveller et al, 2015) despite the recognition that social relations lie 'at the heart of any social intervention' (Poland et al, 2008, p. 301).…”

Meta-Analysis of Complex Interventions

“…As public health researchers and practitioners increasingly embrace (rather than ignore) complexity perspectives, several distinct methodological challenges arise when conducting meta-analyses of intervention effects (26). In addition to questions about whether and to what magnitude an intervention yielded intended effects, consideration of complexity warrants questions regarding for whom and in which contexts interventions are more (or less) effective, which intervention components are most (or least) essential, and how and why the intervention yielded effects (or not).…”