Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death

Abstract

BackgroundPhysician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA–4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method is that it requires a training data set from a prior validation study, while InterVA relies on clinically specified conditional probabilities. We undertook to validate the hierarchical expert algorithm analysis of VA data, an automated, intuitive, deterministic method that does not require a training data set.MethodsUsing Population Health Metrics Research Consortium study hospital source data, we compared the primary causes of 1629 neonatal and 1456 1–59 month–old child deaths from VA expert algorithms arranged in a hierarchy to their reference standard causes. The expert algorithms were held constant, while five prior and one new “compromise” neonatal hierarchy, and three former child hierarchies were tested. For each comparison, the reference standard data were resampled 1000 times within the range of cause–specific mortality fractions (CSMF) for one of three approximated community scenarios in the 2013 WHO global causes of death, plus one random mortality cause proportions scenario. We utilized CSMF accuracy to assess overall population–level validity, and the absolute difference between VA and reference standard CSMFs to examine particular causes. Chance–corrected concordance (CCC) and Cohen’s kappa were used to evaluate individual–level cause assignment.ResultsOverall CSMF accuracy for the best–performing expert algorithm hierarchy was 0.80 (range 0.57–0.96) for neonatal deaths and 0.76 (0.50–0.97) for child deaths. Performance for particular causes of death varied, with fairly flat estimated CSMF over a range of reference values for several causes. Performance at the individual diagnosis level was also less favorable than that for overall CSMF (neonatal: best CCC = 0.23, range 0.16–0.33; best kappa = 0.29, 0.23–0.35; child: best CCC = 0.40, 0.19–0.45; best kappa = 0.29, 0.07–0.35).ConclusionsExpert algorithms in a hierarchy offer an accessible, automated method for assigning VA causes of death. Overall population–level accuracy is similar to that of more complex machine learning methods, but without need for a training data set from a prior validation study.

Social determinants of child mortality in Niger: Results from the 2012 National Verbal and Social Autopsy Study

“…Of note, the government of Niger and stakeholders are engaged in steps for the use of the overall VASA study results to support the revision of the child survival strategy for the country. For example, in order to improve access to better health services for children, the government approved the transformation of some health posts into health centers with an average of 50 health posts transformed per year by December 2014; and the immunization program is being strengthened with the introduction and widespread provision of the Pneumococcal vaccine in the country [ 39 ].…”

Effect of Mass Treatment with Azithromycin on Causes of Death in Children in Malawi: Secondary Analysis from the MORDOR Trial

“…12 Prediction of cause-specific mortality fractions in a population using VA may be approximately 60-80% accurate, although lower for individual-level diagnosis. 13,14 In settings where more accurate determination of cause of death is not possible, VA may be used to produce probable cause of death data with the caveat of it being an imperfect tool. 15 The two most commonly used VA analysis programs are InterVA 16 and SmartVA.…”

Using propensity scores to estimate the effectiveness of maternal and newborn interventions to reduce neonatal mortality in Nigeria

“…The 19,685 surveyed births in the last five years included 538 neonatal deaths, although 212 (39%) of these were not covered in the VASA study and thus were not examined for underlying cause of death. Among 326 neonatal deaths with estimated cause of death, 171 (52%) were caused by sepsis, diarrhea, tetanus, pneumonia or meningitis, and 74 (23%) were due to IPRE [41]. All causes of death as well as ages at death among neonates are shown in Additional file 3.…”

“…Our analysis has limitations. Verbal autopsy cause assignment is not a perfect estimate for cause of death; however, verbal autopsy is likely more accurate when specifying deaths due to any infectious cause compared to death due to a specific infectious cause [41]. Given the complexities in the pathway to receiving interventions and resources, from the decision to seek care, to the receipt of care and the many factors determining quality of care, it is possible that our results are subject to residual confounding or to confounding due to unmeasured factors.…”