Can surveys of women accurately track indicators of maternal and newborn care? A validity and reliability study in Kenya

Abstract

BackgroundTracking progress on maternal and newborn survival requires accurate information on the coverage of essential interventions. Despite widespread use, most indicators measuring maternal and newborn intervention coverage have not been validated. This study assessed the ability of women delivering in two Kenyan hospitals to recall critical elements of care received during the intrapartum and immediate postnatal period at two time points: hospital discharge and 13–15 months following delivery.MethodsWomen’s reports of received care were compared against observations by trained third party observers. Indicators selected for validation were either currently in use or have the potential to be included in population–based surveys. We used a mixed–methods approach to validate women’s reporting ability. We calculated individual–reporting accuracy using the area under the receiver operating curve (AUC), population–level accuracy using the inflation factor (IF), and compared the accuracy of women’s reporting at baseline and follow–up. We also assessed the consistency of women’s reporting over time. We used in–depth interviews with a sub–set of women (n = 20) to assess their understanding of key survey terms.ResultsOf 606 women who participated at baseline and agreed to follow–up, 515 were re–interviewed. Thirty–eight indicators had sufficient sample size for validation analysis; ten met criteria for high or moderate reporting accuracy (0.60<AUC) alone and ten met criteria for low population–level bias alone (0.75<IF<1.25). There was a significant decline in the individual level reporting accuracy between baseline and follow–up for ten indicators. Seven indicators had moderate or higher (0.4≤rphi) consistency between self–reports at baseline and follow–up. Four indicators met all criteria at follow–up: support person was present during the birth, episiotomy, caesarean section, and low birthweight infant (<2500 g).ConclusionThe few indicators that women reported accurately at baseline were consistently recalled with accuracy at 13–15 months follow–up. Although there is deterioration in women’s recall in some indicators over time, the extent of deterioration does not appreciably compromise reporting accuracy for indicators with high baseline validity. Indicators related to initial client assessment and the immediate postnatal period have generally low accuracy and poor reporting consistency over time.

Reliability of maternal recall of delivery and immediate newborn care indicators in Sarlahi, Nepal

“…However, the kappa values for most interventions were low, suggesting observed agreement was mostly due to chance from very high or very low intervention Mexico and Kenya, which showed poor recall of peripartum interventions among women at initial discharge from their labor and delivery facility (2,3). Similar to previous studies, we found recall close to the time of care was generally poor; however, there was little evidence of reliability-altering recall degradation with increasing time since delivery up to a two-year recall period (5,6). The DHS asks women to report on delivery and newborn care for their most recent birth in the previous five years;…”

“Every Newborn-BIRTH” protocol: observational study validating indicators for coverage and quality of maternal and newborn health care in Bangladesh, Nepal and Tanzania

“…EN-BIRTH is the first large study to assess validity of newborn and maternal care indicators in routine data systems, doing so at very large scale (>20 000 observed births) across three countries with a high-burden of mortality. Previous maternal and newborn indicator validation studies have focused on testing the validity of women' s self-report method, used in population-based household surveys [25,[46][47][48]64].…”

“…Comparison of the reported indicator to an external data source "gold standard" is recommended [45]. Previous validation studies have mainly focused on population-based intervention coverage indicators for use in household surveys [25,[46][47][48]. Observational studies to determine accuracy of facility registers in high burden settings have typically focused on outcome indicators [29].…”

“…Specificity of routine data will be assessed for those indicators with true negatives and confidence intervals will be computed. "Area Under the Curve" previously used for coverage indicators validation will be used for indicators with true negatives [25,[46][47][48]64].…”

Discordance in postnatal care between mothers and newborns: Measurement artifact or missed opportunity?

“…However, there remains a substantial confusion between intrapartum care and PNC visits [13]. Observation-based validation studies in Kenya and Swaziland have also demonstrated issues with women' s recall of care received during postnatal period [14,15].…”