When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study

Abstract

IntroductionIn critical care observational studies, when clinicians administer different treatments to sicker patients, any treatment comparisons will be confounded by differences in severity of illness between patients. We sought to investigate the extent that observational studies assessing treatments are at risk of incorrectly concluding such treatments are ineffective or even harmful due to inadequate risk adjustment.MethodsWe performed Monte Carlo simulations of observational studies evaluating the effect of a hypothetical treatment on mortality in critically ill patients. We set the treatment to have either no association with mortality or to have a truly beneficial effect, but more often administered to sicker patients. We varied the strength of the treatment’s true effect, strength of confounding, study size, patient population, and accuracy of the severity of illness risk-adjustment (area under the receiver operator characteristics curve, AUROC). We measured rates in which studies made inaccurate conclusions about the treatment’s true effect due to confounding, and the measured odds ratios for mortality for such false associations.ResultsSimulated observational studies employing adequate risk-adjustment were generally able to measure a treatment’s true effect. As risk-adjustment worsened, rates of studies incorrectly concluding the treatment provided no benefit or harm increased, especially when sample size was large (n = 10,000). Even in scenarios of only low confounding, studies using the lower accuracy risk-adjustors (AUROC < 0.66) falsely concluded that a beneficial treatment was harmful. Measured odds ratios for mortality of 1.4 or higher were possible when the treatment’s true beneficial effect was an odds ratio for mortality of 0.6 or 0.8.ConclusionsLarge observational studies confounded by severity of illness have a high likelihood of obtaining incorrect results even after employing conventionally “acceptable” levels of risk-adjustment, with large effect sizes that may be construed as true associations. Reporting the AUROC of the risk-adjustment used in the analysis may facilitate an evaluation of a study’s risk for confounding.Electronic supplementary materialThe online version of this article (doi:10.1186/s13054-015-0923-8) contains supplementary material, which is available to authorized users.

Association of Intensive Care Unit Admission With Mortality Among Older Patients With Pneumonia

“…35,37 Yet, such studies fail to fully address the confounding by indication for ICU admission. 38 For example, many individuals are denied admission to the ICU for reasons that cannot be measured by administrative data or because they do not require life-sustaining therapies, potentially gaining less additional benefit from ICU-level care that cannot fully be accounted for using severity of illness measures. This study addresses the potential for unmeasured confounding with instrumental variable analyses.…”

Association between Initial Fluid Choice and Subsequent In-hospital Mortality during the Resuscitation of Adults with Septic Shock

“…This model, including patient and hospital demographics, comorbidities, cotreatments, fluid volume, and fluid type as independent variable with in-hospital mortality as the dependent variable, had a c-statistic (area under receiver operating characteristic curve) of 0.763 (95% CI, 0.758 to 0.767), which is consistent with sufficiently accurate risk adjustment so as to provide reasonable protection against confounding. 22 In pairwise PSM comparisons, the receipt of balanced crystalloids by hospital day 2 was consistently associated with lower mortality whether colloids were used (relative risk, 0.84; 95% CI, 0.76 to 0.92) or not (relative risk, 0.78; 95% CI, 0.70 to 0.89; fig. 2).…”

ICU Use and Quality of Care for Patients With Myocardial Infarction and Heart Failure

“…15,16 Second, risk adjustment was performed by using available administrative claims, which, if inadequate, may have introduced bias. 34 For instance, administrative claims may miss important clinical variables or imperfectly capture severity of illness; however, we used risk adjustment models similar to those used by the Centers for Medicare & Medicaid Services. Third, we were unable to determine the time between admission to the hospital and to the ICU, a factor that may provide additional insight toward why the ICU is being utilized.…”