Stability of healthcare quality measures for maternal and child services: Analysis of the continuous service provision assessment of health facilities in Senegal, 2012–2018

Abstract

Objective High‐quality healthcare is essential to ensuring maternal and newborn survival. Efficient measurement requires knowing how long measures of quality provide consistent insight for intended uses. Methods We used a repeated health facility assessment in Senegal to calculate structural and process quality of antenatal care (ANC), delivery and child health services in facilities assessed 2 years apart. We tested agreement of quality measures within facilities and regions. We estimated how much input‐adjusted and process quality‐adjusted coverage measures changed for each service when calculated using quality measurements from the same facilities measured 2 years apart. Results Over 6 waves of continuous surveys, 628 paired assessments were completed. Changes at the facility level were substantial and often positive, but inconsistent. Structural quality measures were moderately correlated (0.40–0.69) within facilities over time, more so in hospitals; correlation was <0.20 for process measures based on direct observation of ANC and child visits. Most measures were more strongly correlated once averaged to regions; process quality of child services was not (−0.32). Median relative difference in national‐adjusted coverage estimates was 6.0%; differences in subnational estimates were largest for process quality of child services (19.6%). Conclusion Continuous measures of structural quality demonstrated consistency at regional levels and in higher level facilities over 2 years; results for process measures were mixed. Direct observation of child visits provided inconsistent measures over time. For other measures, linking population data with health facility assessments from up to 2 years prior is likely to introduce modest measurement error in adjusted coverage estimates.

A methodological framework to assess temporal trends and sub-national disparities in healthcare quality metrics using facility surveys, with applications to sick-child care in Kenya, Senegal, and Tanzania

“…Our framework provides a tool to identify the sources of variability in each readiness and 16 ; yet in the present study, we found substantial random temporal variations in both process and readiness of care metrics when considering all facilities assessed in each wave, which may reflect both sampling variability and true changes in the quality metrics.…”

“…Practically, the complete time-series of these modelled healthcare quality metrics could be used to improve subnational estimates of effective coverage. To date, effective coverage-a key metric to monitor progress towards universal health coverage in LMICs (Ng et al ., 2014; Leslie et al ., 2017a), has been calculated by combining estimates from health facility and household surveys, potentially conducted in different years (Carter et al ., 2021); however, this approach can prove problematic if healthcare quality changes rapidly over time (Kruk et al ., 2017; Leslie et al ., 2022). Yearly subnational estimates of healthcare quality metrics mitigate this problem by computing effective coverage from temporally aligned facility and population estimates, with associated uncertainty.…”

“…Estimated average adherence to IMCI diagnostic protocols exceeded 50% for only a few sub-national units across Kenya, Senegal, and Tanzania, in 2020. Recent studies conducted in sub-Saharan Africa similarly reported low levels of compliance with IMCI guidelines (Gage et al ., 2017; Kruk et al ., 2017; Macarayan et al ., 2018; Getachew et al ., 2020; Perales et al ., 2020; Leslie et al ., 2022). For instance, using facility data collected in the Republic of Congo, Cameroon and the Central African Republic, Perales and colleagues found that providers had completed 12 of the 15 IMCI diagnostic protocols in only 8% of sick-child visits (Perales et al ., 2020), a recent analysis of the 2014 Ethiopian SPA found that the three main symptoms of child diseases were assessed in 51% of consultations (Getachew et al ., 2020), and a nine-country analysis of SPA surveys (Haiti, Kenya, Malawi, Namibia, Nepal, Rwanda, Senegal, Tanzania, and Uganda) found that performance of new graduates for sick-child care was low (i.e., an average 0.43 out of 10 recommended actions were observed); all together, these analyses point to a need for strengthening pre-service education and supervision (Lewis et al ., 2019).…”

“…Conversely, a regularly conducted survey with smaller samples would be best suited to estimate metrics displaying substantial temporal but lower spatial variability. A previous study building upon the repeated assessment of the same facilities over two rounds of the SPA survey (2013/2014-2015/2016) in Senegal found greater variations over time in process quality compared to readiness of care metrics (Leslie et al ., 2022); yet in the present study, we found substantial random temporal variations in both process and readiness of care metrics when considering all facilities assessed in each wave, which may reflect both sampling variability and true changes in the quality metrics.…”

“…Yet, most analyses to date are restricted to the most recent survey and focus on cross-sectional analyses to provide national-level estimates of healthcare availability, readiness, and quality. Novel research has compared country-level estimates from several years of facility surveys (Nguhiu et al ., 2017; Cavallaro et al ., 2018; Leslie et al ., 2022), but differences in sample size and design can limit their actual comparability. Statistical modelling frameworks offer means by which such differences can be explicitly accounted for and support trend estimation in readiness and process quality indicators over time.…”