External validation of a mobile clinical decision support system for diarrhea etiology prediction in children: A multicenter study in Bangladesh and Mali

Abstract

Background: Diarrheal illness is a leading cause of antibiotic use for children in low- and middle-income countries. Determination of diarrhea etiology at the point-of-care without reliance on laboratory testing has the potential to reduce inappropriate antibiotic use. Methods: This prospective observational study aimed to develop and externally validate the accuracy of a mobile software application ('App') for the prediction of viral-only etiology of acute diarrhea in children 0-59 months in Bangladesh and Mali. The App used a previously derived and internally validated model consisting of patient-specific ('present patient') clinical variables (age, blood in stool, vomiting, breastfeeding status, and mid-upper arm circumference) as well as location-specific viral diarrhea seasonality curves. The performance of additional models using the 'present patient' data combined with other external data sources including location-specific climate, data, recent patient data, and historical population-based prevalence were also evaluated in secondary analysis. Diarrhea etiology was determined with TaqMan Array Card using episode-specific attributable fraction (AFe) >0.5. Results: Of 302 children with acute diarrhea enrolled, 199 had etiologies above the AFe threshold. Viral-only pathogens were detected in 22% of patients in Mali and 63% in Bangladesh. Rotavirus was the most common pathogen detected (16% Mali; 60% Bangladesh). The present patient + viral seasonality model had an AUC of 0.754 (0.665-0.843) for the sites combined, with calibration-in-the-large α=-0.393 (-0.455 - -0.331) and calibration slope β=1.287 (1.207 - 1.367). By site, the present patient + recent patient model performed best in Mali with an AUC of 0.783 (0.705 - 0.86); the present patient + viral seasonality model performed best in Bangladesh with AUC 0.710 (0.595 - 0.825). Conclusion: The App accurately identified children with high likelihood of viral-only diarrhea etiology. Further studies to evaluate the App's potential use in diagnostic and antimicrobial stewardship are underway. Funding: Funding for this study was provided through grants from the Bill and Melinda Gates Foundation (OPP1198876) and the National Institute of Allergy and Infectious Diseases (R01AI135114). Several investigators were also partially supported by a grant from the National Institute of Diabetes and Digestive and Kidney Diseases (R01DK116163). This investigation was also supported by the University of Utah Population Health Research (PHR) Foundation, with funding in part from the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002538. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in the study design, data collection, data analysis, interpretation of data, or in the writing or decision to submit the manuscript for publication.

Improving Antibiotic Stewardship for Diarrheal Disease With Probability-Based Electronic Clinical Decision Support

“…We used a molecular diagnostic on stool samples as a gold standard to show a discriminatory performance area under the curve = 0.75, calibration-in-the-large α = −0.393, and calibration slope β = 1.287. 7,9 The discriminative performance was comparable with that achieved in external validation of other commonly used prediction models for etiology of infection. 10,11 In the current study, we incorporated the DEP algorithm into a smartphone-based eCDS tool for physicians to use in clinical encounters with pediatric patients with diarrhea and make informed decisions about antibiotic use.…”

“…Our team previously developed a diarrheal etiologic prediction (DEP) algorithm, based on a modular aggregation of statistical models from a large multicenter study of pediatric diarrhea to predict the probability that a patient has a viral etiology of diarrheal illness . The DEP draws upon data from clinical history and symptoms of the patient (patient-specific sources) and location-specific sources, such as clinical presentation of prior patients, historical prevalence, and weather parameters . The performance of the DEP algorithm was assessed in an external validation study at sites in Mali and Bangladesh.…”

“…Second, we did not conduct diagnostic stool tests to verify etiologic predictions; however, the DEP algorithm was previously internally and externally validated at the same or similar study locations. 8,9 Third, given that diarrheal disease-causing pathogens are often seasonal, and our study was conducted over a period of 9 weeks, it is possible that seasonality affected the study periods and that physicians were seeing patients with different characteristics between periods. However, after accounting for the predicted value of the DEP, there was no evidence for a period interacting with treatment effect.…”

“…7,8 The DEP draws upon data from clinical history and symptoms of the patient (patient-specific sources) and location-specific sources, such as clinical presentation of prior patients, historical prevalence, and weather parameters. 7,9 The performance of the DEP algorithm was assessed in an external validation study at sites in Mali and Bangladesh. We used a molecular diagnostic on stool samples as a gold standard to show a discriminatory performance area under the curve = 0.75, calibration-in-the-large α = −0.393, and calibration slope β = 1.287.…”

Designing a Novel Clinician Decision Support Tool for the Management of Acute Diarrhea in Bangladesh: Formative Qualitative Study

“…The algorithms of both models were then incorporated into a mobile app prototype. The prototype was derived from an mHealth CDS ( Rehydration Calculator ) that adapted paper-based WHO guidelines to the digital medium [ 27 , 28 ]. The prototype allowed for clinicians to enter a patient’s symptoms in the input screen ( Figure 1 A) and to receive the patient’s dehydration severity level and specific treatment recommendations on the output screen ( Figure 1 B).…”