Applying a complex adaptive systems approach to the evaluation of a school-based intervention for intimate partner violence prevention in Mexico

Abstract

Despite calls for evaluation practice to take a complex systems approach, there are few examples of how to incorporate complexity into real-life evaluations. This article presents the case for using a complex systems approach to evaluate a school-based intimate partner violence-prevention intervention. We conducted a post hoc analysis of qualitative evaluation data to examine the intervention as a potential system disruptor. We analysed data in relation to complexity concepts particularly relevant to schools: ‘diverse and dynamic agents’, ‘interaction’, ‘unpredictability’, ‘emergence’ and ‘context dependency’. The data—two focus groups with facilitators and 33 repeat interviews with 14–17-year-old students—came from an evaluation of a comprehensive sexuality education intervention in Mexico City, which serves as a case study for this analysis. The findings demonstrate an application of complex adaptive systems concepts to qualitative evaluation data. We provide examples of how this approach can shed light on the ways in which interpersonal interactions, group dynamics, the core messages of the course and context influenced the implementation and outcomes of this intervention. This gender-transformative intervention appeared to disrupt pervasive gender norms and reshape beliefs about how to engage in relationships. An intervention comprises multiple dynamic and interacting elements, all of which are unlikely to be consistent across implementation settings. Applying complexity concepts to our analysis added value by helping reframe implementation-related data to focus on how the ‘social’ aspects of complexity influenced the intervention. Without examining both individual and group processes, evaluations may miss key insights about how the intervention generates change, for whom, and how it interacts with its context. A social complex adaptive systems approach is well-suited to the evaluation of gender-transformative interventions and can help identify how such interventions disrupt the complex social systems in which they are implemented to address intractable societal problems.

Evaluating Complex Interventions Using Qualitative Longitudinal Research: A Case Study of Understanding Pathways to Violence Prevention

“…Analyzing case histories from repeat interview data ( Henderson et al, 2012 ; Thomson, 2007 ) allowed us to link participants’ individual circumstances and context with their experiences in, and responses to, the intervention. This helped explain variability in how different participants experienced the intervention ( Makleff, Billowitz, et al, 2020 ). For example, in Cases 1, 2, and 4, participants reflected on their family situation in relation to the course contents or described discussing the intervention with family members—suggesting that violence-related beliefs and experiences at home and intervention experiences have mutual influence.…”

“…We also conducted focus group discussions with teachers and health educators, and observed the intervention classroom sessions, recording observations using field notes. More details about the course activities, study setting, research design, data collection, and evaluation findings are published elsewhere ( Makleff, Billowitz, et al, 2020 ; Makleff, Garduño, et al, 2020 ). This article focuses on the data gathered using repeat interviews, and draws on complementary information gathered using other data collection methods when necessary to provide context.…”

Collecting better data on sexuality, relationships and violence: empirical evidence from a school-based evaluation

“…We have described further details of the intervention and its evaluation elsewhere (33,34). The intervention was developed by Mexfam and took place in one secondary school (preparatoria) in the south of Mexico City with students between 14 and 17 years of age.…”

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

“…47 A complex systems perspective applied to public health evaluation involves analysing the multiple ways in which a complex system and an intervention interact and influence each other to generate health impacts and health inequalities. [48][49][50][51] Evaluators might consider interventions as 'events' within systems that have the ability to disrupt system behaviour, generating evolving and adaptive patterns of behaviour and emergent outcomes. 44 52 In public health, process evaluations have traditionally been used to understand the mechanisms by which interventions lead to impact, the influence of the broader context on observed variations in impact as well as to assess intervention fidelity and the quality of implementation.…”