From raw data to a score: Comparing quantitative methods that construct multi-level composite implementation strength scores of family planning programs in Malawi

Abstract

Background: Data that capture implementation strength can be combined in multiple ways across content and health system levels to create a summary measure that can help us to explore and compare program implementation across facility catchment areas. Summary indices can make it easier for national policymakers to understand and address variation in strength of program implementation across jurisdictions. In this paper we describe development of an index that we used to describe the district level strength of implementation of the Malawi national family planning program. Methods: To develop the index, we used data collected during a 2017 national, health facility and community health worker Implementation Strength Assessment survey in Malawi to test different methods to combine indicators within and then across domains (4 methods: simple additive, weighted additive, principal components analysis, exploratory factor analysis) and combine scores across health facility and community health worker levels (2 methods: simple average and mixed effects model) to create a catchment area-level summary score for each health facility in Malawi. We explored how well each model captures variation and predicts couple years protection and how feasible it is to conduct each type of analysis and the resulting interpretability. Results: We found little difference in how the four methods combined indicator data at the individual and combined levels of the health system. However, there were major differences when combining scores across health system levels to obtain a score at the health facility catchment area level. The scores resulting from the mixed effects model were able to better discriminate differences between catchment area scores compared to the simple average method. The scores using the mixed effects combination method also demonstrated more of a dose and response relationship with couple years protection. Conclusions: The summary measure that was calculated from the mixed effects combination method captured the variation of strength of implementation of the Malawi national family planning program at the health facility catchment area level. However, the best method for creating an index should be based on pros and cons listed, not least, analyst capacity and ease of interpretability of findings. Ultimately, the resulting summary measure can aid decisionmakers in understanding the combined effect of multiple aspects of programs being implemented in their health system and comparing strengths of programs across geographies.

The aggregate effect of implementation strength of family planning programs on modern contraceptive use at the health systems level in rural Malawi

“…There have been mixed results among studies aiming to demonstrate a link between structural quality (an analogue to IS) and contraceptive use [ 22 – 25 , 29 , 30 , 41 ]. Most of these studies are restricted to health facility data (often from Service Provision Assessments) and do not take outreach services into account like this study did.…”

“…Additionally, this study fills a gap by analyzing how the combined strength of several FP programs being implemented can have an impact on modern contraceptive use. Most other studies we reviewed evaluate either single, specific programs or the readiness of health facilities (either without including outreach or analyzing it separately) [ 29 , 30 , 40 ].…”

“…We categorized the summary measure into quartiles for easier interpretation and comparison. More detail about the creation of the composite IS score can be found in a previous study by Pattnaik et al [ 30 ]. This includes why the PCA and MEM methods were chosen and results from the statistical regression methods used to create the implementation strength scores.…”

“…Previous studies have shown mixed results in demonstrating a clear association between structural quality or readiness (an analogue to implementation strength) and contraceptive use [ 22 – 24 ]. Yet, very few of these studies include outreach services or analyze the combined impact of the implementation of multiple FP programs at the national scale [ 25 , 26 , 30 ]. We conducted an Implementation Strength Assessment (ISA) in 2017 to measure the strength of FP programs delivered across the community and facility levels in Malawi and developed a summary measure that reflects the combined implementation strength of FP programs across these levels [ 27 ].…”