Naive Bayes classifiers for verbal autopsies: comparison to physician-based classification for 21,000 child and adult deaths

Abstract

Background Verbal autopsies (VA) are increasingly used in low- and middle-income countries where most causes of death (COD) occur at home without medical attention, and home deaths differ substantially from hospital deaths. Hence, there is no plausible “standard” against which VAs for home deaths may be validated. Previous studies have shown contradictory performance of automated methods compared to physician-based classification of CODs. We sought to compare the performance of the classic naive Bayes classifier (NBC) versus existing automated classifiers, using physician-based classification as the reference. Methods We compared the performance of NBC, an open-source Tariff Method (OTM), and InterVA-4 on three datasets covering about 21,000 child and adult deaths: the ongoing Million Death Study in India, and health and demographic surveillance sites in Agincourt, South Africa and Matlab, Bangladesh. We applied several training and testing splits of the data to quantify the sensitivity and specificity compared to physician coding for individual CODs and to test the cause-specific mortality fractions at the population level. Results The NBC achieved comparable sensitivity (median 0.51, range 0.48-0.58) to OTM (median 0.50, range 0.41-0.51), with InterVA-4 having lower sensitivity (median 0.43, range 0.36-0.47) in all three datasets, across all CODs. Consistency of CODs was comparable for NBC and InterVA-4 but lower for OTM. NBC and OTM achieved better performance when using a local rather than a non-local training dataset. At the population level, NBC scored the highest cause-specific mortality fraction accuracy across the datasets (median 0.88, range 0.87-0.93), followed by InterVA-4 (median 0.66, range 0.62-0.73) and OTM (median 0.57, range 0.42-0.58). Conclusions NBC outperforms current similar COD classifiers at the population level. Nevertheless, no current automated classifier adequately replicates physician classification for individual CODs. There is a need for further research on automated classifiers using local training and test data in diverse settings prior to recommending any replacement of physician-based classification of verbal autopsies.

Generalized Bayes Quantification Learning under Dataset Shift

“…The number of observations within India, Mexico, Philippines, and Tanzania are 2973, 1586, 1259, and 2023, respectively. To address countryspecific dataset shift, for each country, we used the three remaining countries as training data for four methods commonly used for cause of death predictions: InterVA (Byass et al, 2012), InSilicoVA (McCormick et al, 2016, NBC (Miasnikof et al, 2015), and Tariff (Serina et al, 2015). The first three methods are probabilistic, while Tariff produces a score for each cause that needed to be normalized to be in [0, 1].…”

“…However, in some applications, the objective is not individual level predictions, but rather to learn about population-level distributions of a given outcome. Examples include sentiment analysis for Twitter users (Giachanou and Crestani, 2016), estimating the prevalence of chronic fatigue syndrome (Valdez et al, 2018), and cause of death distribution estimation from verbal autopsies (King et al, 2008;McCormick et al, 2016;Serina et al, 2015;Byass et al, 2012;Miasnikof et al, 2015).…”

Verbal Autopsy in Civil Registration and Vital Statistics: The Symptom-Cause Information Archive

“…[15] Physician-coding of VA (PCVA) is subject to physician-specific bias, occupies valuable physician time, is expensive, and often leads to long delays between the VA interview and assigning a cause of death. Over the past decade computational algorithms [16,17,18,19,20,21,22,23] have been developed and are increasingly used to interpret VA data and assign causes to VA deaths. Algorithms are consistent, essentially cost-free (compared to the traditional use of physician time to assign cause of death to VA), and can be run on large numbers of deaths quickly.…”

“…In some cases SCI take the form of deaths with symptoms and causes assigned through another mechanism, and in other cases, the relationships are solicited directly from experts. There are six VA-coding algorithms that have been proposed and/or used widely: (i) InterVA, [26,20,27,28,29,30] (ii) Tariff, [21] (iii) a derivative of Tariff called SmartVA-Analyze, [18,19] (iv) InSilicoVA, [16,31] (v) a naive Bayes classifier called NBC, [17] and (vi) the King-Lu algorithm. [22,23] The list of causes that each algorithm assigns varies slightly.…”

Bayesian Hierarchical Factor Regression Models to Infer Cause of Death From Verbal Autopsy Data

“…Existing computer-coded VA algorithms include those for which the relationship between symptoms and cause of death is encoded by experts (InterVA [Byass et al, 2012] and InSilicoVA [McCormick et al, 2016]) and those for which it is learned by relying on a labeled subset of the data having known COD (the King and Lu method [King et al, 2008], the Tariff method [James et al, 2011], the Simplified Symptom Pattern method [Murray et al, 2011a], the naive Bayes classifier [Miasnikof et al, 2015], the Bayesian factor model [Kunihama et al, 2018], and latent Gaussian graphical model [Li et al, 2018b]).…”