How beneficiaries see complex health interventions: a practice review of the Most Significant Change in ten countries

Abstract

Background The Most Significant Change is a story-based evaluation approach used in many international development programs. This practice review summarises practical experience with the approach in complex health interventions in ten countries, with the objective of making it more accessible in evaluation of other complex health interventions. Results Participatory research practitioners and trainees discussed five themes following brief presentations by each of the seven attendees who led the exercise: (i) sampling and recruitment; (ii) phrasing the questions to elicit stories; (iii) story collection strategies; (iv) quality assurance; and (v) analysis. Notes taken during the meeting provided the framework for this article. Recruitment strategies in small studies included universal engagement and, in larger studies, a purposive, systematic or random sampling. Meeting attendees recommended careful phrasing and piloting of the question(s) as this affects the quality and focus of the stories generated. They stressed the importance of careful training and monitoring of fieldworkers collecting stories to ensure full stories are elicited and recorded. For recording, in most settings they preferred note taking with back-checking or self-writing of stories by story tellers, rather than audio-recording. Analysis can combine participatory selection of a small number of stories, deductive or inductive thematic analysis and discourse analysis. Meeting attendees noted that involvement in collection of the stories and their analysis and discussion had a positive impact for research team members. Conclusions Our review confirms the plasticity, feasibility and acceptability of the Most Significant Change technique across different sociopolitical, cultural and environmental contexts of complex interventions. Although the approach can surface unexpected impacts, it is not a 360-degree evaluation. Its strength lies in characterising the changes, where these happen, in the words of the beneficiaries. We hope this distillation of our practice makes the technique more readily available to health sector researchers.

Lessons Learned From the Use of the Most Significant Change Technique for Adaptive Management of Complex Health Interventions

“…Refresher training is essential because programmatic staff are not always comfortable with probing the interviewees to elicit necessary details. 4 It is also beneficial to have a dedicated staff responsible for coordinating MSC implementation and supporting story collectors regularly. Field staff members and local government officials who collect and write stories appreciate having input from the hub teams that review the stories.…”

“…The users of MSC generally report positive experiences with the method, particularly noting unique roles and contributions of the narrative approach. Tonkin et al 4 noted that the strength of the MSC technique lies in characterizing the change in complex health interventions and confirmed its plasticity, feasibility, and acceptability across different sociopolitical, cultural, and environmental contexts. Consequently, the MSC approach would work well to help evaluate the scale-up of evidence-based interventions across multiple countries.…”

Evaluating a peer-led wellbeing programme for doctors-in-training during the COVID-19 pandemic in Victoria, Australia, using the Most Significant Change technique

“…Success is not measured against programme goals; instead, emergent outcomes and value placed on significant changes form the focus of inquiry. MSC is particularly useful for: understanding the nature of change through participants’ voices ( Tonkin et al, 2021 ; Limato et al, 2018 ), identifying unintended programme outcomes ( Willetts & Crawford, 2007 ) and enabling stakeholders to articulate and focus on what they find most important ( Shah, 2014 ). Basic steps, referred to as ‘the data cycle’, include identification of required data, collection of stories of change, selection of a most significant change story (or stories), analysis of key themes and communication of outcomes ( Dart & Davies, 2003 ; Willetts & Crawford, 2007 ).…”

“…Stakeholders select one story exemplifying the most significant change resulting from the evaluated program. Inductive thematic analysis of all stories is combined with reasons for making the selection, to inform learnings ( Dart & Davies, 2003 ; Tonkin et al, 2021 ). Nine stories of change were included in the selection.…”

“I had to change my attitude”: narratives of most significant change explore the experience of universal home visits to pregnant women and their spouses in Bauchi State, Nigeria

“…Although MSC is widely used in evaluation of development projects it has been much less used in the evaluation of health interventions. A recent report summarized health sector experience with using MSC in participatory initiatives [35].…”