Design of an impact evaluation using a mixed methods model – an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi

Abstract

BackgroundIn this article we present a study design to evaluate the causal impact of providing supply-side performance-based financing incentives in combination with a demand-side cash transfer component on equitable access to and quality of maternal and neonatal healthcare services. This intervention is introduced to selected emergency obstetric care facilities and catchment area populations in four districts in Malawi. We here describe and discuss our study protocol with regard to the research aims, the local implementation context, and our rationale for selecting a mixed methods explanatory design with a quasi-experimental quantitative component.DesignThe quantitative research component consists of a controlled pre- and post-test design with multiple post-test measurements. This allows us to quantitatively measure ‘equitable access to healthcare services’ at the community level and ‘healthcare quality’ at the health facility level. Guided by a theoretical framework of causal relationships, we determined a number of input, process, and output indicators to evaluate both intended and unintended effects of the intervention. Overall causal impact estimates will result from a difference-in-difference analysis comparing selected indicators across intervention and control facilities/catchment populations over time.To further explain heterogeneity of quantitatively observed effects and to understand the experiential dimensions of financial incentives on clients and providers, we designed a qualitative component in line with the overall explanatory mixed methods approach. This component consists of in-depth interviews and focus group discussions with providers, service user, non-users, and policy stakeholders. In this explanatory design comprehensive understanding of expected and unexpected effects of the intervention on both access and quality will emerge through careful triangulation at two levels: across multiple quantitative elements and across quantitative and qualitative elements.DiscussionCombining a traditional quasi-experimental controlled pre- and post-test design with an explanatory mixed methods model permits an additional assessment of organizational and behavioral changes affecting complex processes. Through this impact evaluation approach, our design will not only create robust evidence measures for the outcome of interest, but also generate insights on how and why the investigated interventions produce certain intended and unintended effects and allows for a more in-depth evaluation approach.

Health Insurance and Children in Low- and Middle-Income Countries: A Review

“…Furthermore, the estimation of different treatment effects -local average treatment effect for regression discontinuity (Palmer et al, 2015) versus average treatment effect for the difference-in-difference estimation strategy for the other two papers -may have contributed to the different findings. These different results for Vietnam's programs contrast with those from a review of the literature in the United States where studies with different methods lead to similar results (Buchmueller et al, 2005).…”

“…For instance, regarding inpatient care, we found several studies that examine the effects of insurance on any hospital utilisation by children. However, these studies do not differentiate between avoidable and unavoidable inpatient care, as has sometimes been done in the literature in high-income countries (Buchmueller et al, 2005). Some of the mixed results on the impact of health insurance on care utilisation and health outcomes are perhaps due to the limited information available in standard household surveys.…”

“…The expected effect is less clear for inpatient visits due to increased preventative and primary health-care usage. Buchmueller et al (2005) claim that a lack of insurance may cause inefficient use of medical care and avoidable hospitalisations (inadequate prevention and excessive reliance on inpatient care). Expanding health insurance coverage may thus lead to reductions in some types of hospital utilisation.…”

Perceptions of quality across the maternal care continuum in the context of a health financing intervention: Evidence from a mixed methods study in rural Malawi

“…The evaluation relied on a mixed-methods prospective sequential controlled pre- and post-test design with independent controls, whereby we integrated quantitative and qualitative methods of data collection and analysis within a single design [39]. The choice of study design was determined in light of the fact that the intervention facilities were purposely (and not randomly) selected by the MoH among the complete set of EmOC facilities in the four concerned districts.…”

Health Insurance and Children in Low‐ and Middle‐income Countries: A Review

“…Bolivia established its National Maternal and Child Insurance before expanding it into a national public health insurance program. Similarly, Malawi introduced maternal and newborn health insurance in 2004 in response to the MDGs (Brenner et al, 2014). Several reviews of the impact of health insurance exist, but none specific to children (e.g.…”