Application of Balanced Scorecard in the Evaluation of a Complex Health System Intervention: 12 Months Post Intervention Findings from the BHOMA Intervention: A Cluster Randomised Trial in Zambia

Abstract

IntroductionIn many low income countries, the delivery of quality health services is hampered by health system-wide barriers which are often interlinked, however empirical evidence on how to assess the level and scope of these barriers is scarce. A balanced scorecard is a tool that allows for wider analysis of domains that are deemed important in achieving the overall vision of the health system. We present the quantitative results of the 12 months follow-up study applying the balanced scorecard approach in the BHOMA intervention with the aim of demonstrating the utility of the balanced scorecard in evaluating multiple building blocks in a trial setting.MethodsThe BHOMA is a cluster randomised trial that aims to strengthen the health system in three rural districts in Zambia. The intervention aims to improve clinical care quality by implementing practical tools that establish clear clinical care standards through intensive clinic implementations. This paper reports the findings of the follow-up health facility survey that was conducted after 12 months of intervention implementation. Comparisons were made between those facilities in the intervention and control sites. STATA version 12 was used for analysis.ResultsThe study found significant mean differences between intervention(I) and control (C) sites in the following domains: Training domain (Mean I:C; 87.5.vs 61.1, mean difference 23.3, p = 0.031), adult clinical observation domain (mean I:C; 73.3 vs.58.0, mean difference 10.9, p = 0.02 ) and health information domain (mean I:C; 63.6 vs.56.1, mean difference 6.8, p = 0.01. There was no gender differences in adult service satisfaction. Governance and motivation scores did not differ between control and intervention sites.ConclusionThis study demonstrates the utility of the balanced scorecard in assessing multiple elements of the health system. Using system wide approaches and triangulating data collection methods seems to be key to successful evaluation of such complex health intervention.Trial numberClinicalTrials.gov NCT01942278

Application of systems thinking: 12‐month postintervention evaluation of a complex health system intervention in Zambia: the case of the BHOMA

“…We report both intended and unintended consequences and how the context affected the results of the intervention. The quantitative results are reported in another paper [27].…”

Management Matters: A Leverage Point for Health Systems Strengthening in Global Health

“…Although management has been recognized as fundamental to health worker performance and motivation, [14][15][16] peer-reviewed studies on management, including quality improvement efforts, in global health are limited in their size, scope, and rigor. About 2 dozen studies [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37] including a literature review 38 have examined the link between management and health system performance in low-and middle-income settings in the last 20 years. These studies are largely uniform in their findings, demonstrating that interventions to build management capacity in facilities and district health teams can influence performance.…”

Associations between health systems capacity and mother-to-child HIV prevention program outcomes in Zambia

“…The patient satisfaction domain comprised adult satisfaction and child satisfaction indices, which were assessed by exit interviews among five adults and five under-five child/guardian pairs, respectively[ 16 , 18 ]. Interviewers asked participants to rate each of the following on a 5-point scale: wait time to be seen by a provider, the provider’s explanation of illness and treatment, and the perceived appropriateness of the treatment received.…”

“…Our unit of analysis was the health facility and its surrounding community (also referred to as the “site”). Health system capacity, our primary exposure, was evaluated using the balanced scorecard instrument described above[ 16 , 18 ]. Domain-specific scores were calculated for patient satisfaction, human resources, finance, governance, service capacity, and service provision.…”