Evaluating complex interventions is complicated. The Medical Research Council's evaluation framework (2000) brought welcome clarity to the task. Now the council has updated its guidance

No abstract

The UK Medical Research Council’s widely used guidance for developing and evaluating complex interventions has been replaced by a new framework, commissioned jointly by the Medical Research Council and the National Institute for Health Research, which takes account of recent developments in theory and methods and the need to maximise the efficiency, use, and impact of research.

Natural experimental studies are often recommended as a way of understanding the health impact of policies and other large scale interventions. Although they have certain advantages over planned experiments, and may be the only option when it is impossible to manipulate exposure to the intervention, natural experimental studies are more susceptible to bias. This paper introduces new guidance from the Medical Research Council to help researchers and users, funders and publishers of research evidence make the best use of natural experimental approaches to evaluating population health interventions. The guidance emphasises that natural experiments can provide convincing evidence of impact even when effects are small or take time to appear. However, a good understanding is needed of the process determining exposure to the intervention, and careful choice and combination of methods, testing of assumptions and transparent reporting is vital. More could be learnt from natural experiments in future as experience of promising but lesser used methods accumulates

BackgroundThere are concerns that COVID-19 mitigation measures, including the ‘lockdown’, may have unintended health consequences. We examined trends in mental health and health behaviours in the UK before and during the initial phase of the COVID-19 lockdown and differences across population subgroups.MethodsRepeated cross-sectional and longitudinal analysis of the UK Household Longitudinal Study, including representative samples of over 27,000 adults (aged 18+) interviewed in four survey waves between 2015 and 2020. A total of 9748 adults had complete data for longitudinal analyses. Outcomes included psychological distress (General Health Questionnaire-12), loneliness, current cigarette smoking, use of e-cigarettes and alcohol consumption. Cross-sectional prevalence estimates were calculated and multilevel Poisson regression assessed associations between time period and the outcomes of interest, as well as differential associations by age, gender, education level and ethnicity.ResultsPsychological distress increased 1 month into lockdown with the prevalence rising from 19.4% (95% CI 18.7% to 20.1%) in 2017–2019 to 30.6% (95% CI 29.1% to 32.3%) in April 2020 (RR=1.3, 95% CI 1.2 to 1.4). Groups most adversely affected included women, young adults, people from an Asian background and those who were degree educated. Loneliness remained stable overall (RR=0.9, 95% CI 0.6 to 1.5). Smoking declined (RR=0.9, 95% CI=0.8,1.0) and the proportion of people drinking four or more times per week increased (RR=1.4, 95% CI 1.3 to 1.5), as did binge drinking (RR=1.5, 95% CI 1.3 to 1.7).ConclusionsPsychological distress increased 1 month into lockdown, particularly among women and young adults. Smoking declined, but adverse alcohol use generally increased. Effective measures are required to mitigate negative impacts on health.

Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.

time that this paper was being developed and written. Charlotte Loppie reports a grant from the CIHR that funded research reported in a case study in this report. Laurence Moore reports having been a member of the UK MRC Population Health Strategy Group and the MRC/NIHR Methodology Research Programme Panel during the life of this project. He also reports core funding from the MRC and the Scottish Government CSO. David Ogilvie reports a grant from the NIHR Public Health Research programme and a grant from the MRC programme during the life of the project. Mark Petticrew reports a grant from the NIHR to develop a briefing paper. Valéry Ridde reports conducting consultancy work for non-governmental organisations implementing the user fees exemption intervention in West Africa. Daniel Wight reports grants from the UK MRC and the NIHR. Outside the submitted work, he reports core funding from the UK MRC to lead a theme of research on the transferability of interventions.

Background The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. Methods There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments. Results Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six ‘core elements’ that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout. Limitations The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development. Conclusions This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as ‘events in systems’ that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers. Future work We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources. Funding This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).