Statistical Uncertainty of Mortality Rates and Rankings for Children's Hospitals

Abstract

WHAT'S KNOWN ON THIS SUBJECT:Many hospital quality-of-care stakeholders wish to identify hospitals with leading or lagging performance on the basis of mortality rates, but the degree of statistical imprecision of comparative rates or rankings may not be fully appreciated. WHAT THIS STUDY ADDS:Children's hospitals' mortality rates have substantial statistical imprecision, and hospital rankings based on those rates have even more. Stakeholders who seek to improve quality of care and patient outcomes may want to find more informative measures to guide action. abstract OBJECTIVES: Hospitals are being required to report publically their adjusted mortality rates, which are then being used to rank hospitals. Our objectives were to assess the statistical reliability of the determination of a hospital's adjusted mortality rate, of comparisons of that rate with the rates of other hospitals, and of the use of those rates to rank the hospitals. METHODS:A cross-sectional study of 473 383 patients discharged from 42 US children's hospitals in 2008 was performed. Hospitalspecific observed/expected (O/E) mortality rate ratios and corresponding hospital rankings, with 95% confidence intervals (CIs), were examined. RESULTS:Hospitals' O/E mortality rate ratios exhibited wide 95% CIs, and no hospital was clearly distinguishable from the other hospitals' aggregated mean mortality performance. Only 2 hospitals' mortality performance fell outside the comparator hospitals' 95% CI. Those hospitals' 95% CIs overlapped with the overall comparator set's 95% CI, which suggests that there were no statistically significant hospital outliers. Fourteen (33.3%) of the 42 hospitals had O/E ratios that were not statistically different from being in the 95% CI of the top 10% of hospitals. Hospital-specific mortality rate rankings displayed even broader 95% CIs; the typical hospital had a 95% CI range that spanned 22 rank-order positions. CONCLUSION:Children's hospital-specific measures of adjusted mortality rate ratios and rankings have substantial amounts of statistical imprecision, which limits the usefulness of such measures for comparisons of quality of care.

Development and validation of the pediatric risk estimate score for children using extracorporeal respiratory support (Ped-RESCUERS)

“…In addition to primary diagnoses, candidate variables included clinical data collected ≤ 6 hours prior to ECMO, such as physiologic (the worst pre-ECMO blood gas and lowest systolic blood pressure) and therapeutic (ventilator settings and number of days of mechanical ventilation prior to ECMO) data. Other variables included the presence of pre-ECMO cardiac arrest, pre-ECMO renal failure and any clinical comorbidities as defined by Feudtner et al [25] (additional details in Online Resource 1-Supplemental Methods and Table e2). A priori we identified two types of interactions to consider: the interaction between ventilator settings and ventilator type and an interaction between blood pressure and age.…”

“… a Most abnormal value recorded within 6 hours of receipt of ECMO support, b Comorbidities are defined by Feudtner et al [25] c Pre-ECMO renal failure and pre-ECMO cardiac arrest are defined by ICD-9-CM codes plus the respective absence of renal failure or cardiac arrest as an ECMO complication. …”

Limitations of Ranking Lists Based on Cardiac Surgery Mortality Rates

“…Ranstam showed that even small amounts of missing data can considerably increase the margin of error around ranks, and Feudtner found wide confidence intervals around ranks based on mortality rates as well. 13,14 When only 1 certain type of procedure was analyzed (in this case, isolated CABG procedures), results were similar. The average confidence interval was even slightly larger compared with those resulting from all procedures because of the smaller sample sizes.…”

Benchmarking statewide trauma mortality using Agency for Healthcare Research and Quality's patient safety indicators

“…1). This provided the identification of hospitals higher or lower than the expected mortality range set by the overall group mean to a statistically significant degree [9].…”