External validation of the RISC, RISC-Malawi, and PERCH clinical prediction rules to identify risk of death in children hospitalized with pneumonia

Abstract

Background Existing scores to identify children at risk of hospitalized pneumonia-related mortality lack broad external validation. Our objective was to externally validate three such risk scores. Methods We applied the Respiratory Index of Severity in Children (RISC) for HIV-negative children, the RISC-Malawi, and the Pneumonia Etiology Research for Child Health (PERCH) scores to hospitalized children in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set. The PREPARE data set includes pooled data from 41 studies on pediatric pneumonia from across the world. We calculated test characteristics and the area under the curve (AUC) for each of these clinical prediction rules. Results The RISC score for HIV-negative children was applied to 3574 children 0-24 months and demonstrated poor discriminatory ability (AUC = 0.66, 95% confidence interval (CI) = 0.58-0.73) in the identification of children at risk of hospitalized pneumonia-related mortality. The RISC-Malawi score had fair discriminatory value (AUC = 0.75, 95% CI = 0.74-0.77) among 17 864 children 2-59 months. The PERCH score was applied to 732 children 1-59 months and also demonstrated poor discriminatory value (AUC = 0.55, 95% CI = 0.37-0.73). Conclusions In a large external application of the RISC, RISC-Malawi, and PERCH scores, a substantial number of children were misclassified for their risk of hospitalized pneumonia-related mortality. Although pneumonia risk scores have performed well among the cohorts in which they were derived, their performance diminished when externally applied. A generalizable risk assessment tool with higher sensitivity and specificity to identify children at risk of hospitalized pneumonia-related mortality may be needed. Such a generalizable risk assessment tool would need context-specific validation prior to implementation in that setting.

Factors associated with severe childhood community-acquired pneumonia: a retrospective study from two hospitals

“…Therefore, children with pneumonia of the same age as 2 years old who have a respiratory rate of 50 breaths per minute and 70 breaths per minute have the same prognosis, according to the revised WHO classification. Our finding indicates that an increase in respiratory rate contributes to the severity of pneumonia, and this finding has been replicated elsewhere [17,19,20,22,33]. For instance, Agweyu et al [18] find that a respiratory rate of 70 breaths per minute or more is independently associated with death.…”

“…In one study, an axillary temperature of ≥ 39 °C has an adjusted risk ratio of 1.9 for mortality in pediatric pneumonia [18]. Furthermore, the duration of illness, including fever, has been found as a factor that is associated with an adverse outcome in childhood pneumonia [19,20]. A study from Bangladesh finds that symptoms that last more than 3 days increase the risk of adverse outcomes in pediatric CAP [21].…”

Derivation and validation of a novel risk assessment tool to identify children aged 2–59 months at risk of hospitalised pneumonia-related mortality in 20 countries

“…Prior clinical prediction models developed in single countries (ie, RISC, mRISC and RISC-Malawi) have demonstrated AUC from 0.80 to 0.92 when internally validated,4–6 while the PERCH score, derived from five countries in sub-Saharan Africa as well as Thailand and Bangladesh, had an AUC of 0.76 on internal validation 7. However, when externally applied to patients in various settings, only the RISC-Malawi score had fair discriminatory ability 9. The PREPARE risk assessment tool transcends region-specific issues, such as differing epidemiology of causative viruses or bacteria and variations in availability of measures such as chest radiography, supplemental oxygen, antibiotic availability and unmeasured variables that can contribute to morbidity, given its derivation and validation from a widely representative patient population.…”

“… 7 However, when externally applied to patients in various settings, only the RISC-Malawi score had fair discriminatory ability. 9 The PREPARE risk assessment tool transcends region-specific issues, such as differing epidemiology of causative viruses or bacteria and variations in availability of measures such as chest radiography, supplemental oxygen, antibiotic availability and unmeasured variables that can contribute to morbidity, given its derivation and validation from a widely representative patient population. However, the PREPARE risk assessment tool must be externally validated prior to implementation.…”

“…We recently validated three of these risk assessment tools in a large, globally representative dataset using the demographic and clinical features of children at the time of admission. 9 In that external validation, only the Respiratory Index of Severity in Children–Malawi (RISC-Malawi) score demonstrated fair discriminatory value (area under the curve (AUC) 0.75, 95% CI 0.74 to 0.77), while the Respiratory Index of Severity in Children (RISC) score and a modified Pneumonia Etiology Research for Child Health (PERCH) score had limited discriminatory value in identifying hospitalised children at risk of pneumonia-related mortality (AUC 0.66, 95% CI 0.58 to 0.73%, and AUC 0.55, 95% CI 0.37 to 0.73, respectively). The suboptimal performance of these prediction rules when applied externally raises questions on whether a novel tool incorporating different combinations of widely available clinical indicators could perform better across broad settings.…”