Randomized Clinical Trials or Convenient Controls: TREWS or FALSE?

Abstract

We read with interest the Adams et al. (doi: https://doi.org/10.1038/s41591-022-01894-0) report of the TREWS machine learning (ML)-based sepsis early warning system. The authors conclude that large-scale randomized trials are needed to confirm their observations, but assert that their findings indicate the potential for the TREWS system to identify sepsis patients early and improve patient outcomes, including a significant decrease in mortality. However, this conclusion is based upon a comparison of those whose alert was confirmed vs. not confirmed within 3 hours, rather than random allocation to TREWS vs. no TREWS. Using data from over 650,000 patient encounters across two distinct healthcare systems, we show that the findings of Adams et al. are likely to be severely biased due to the failure to adjust for 'processes of care'-related confounding factors.

Addressing the ‘coin flip model’ and the role of ‘process of care’ variables in the analysis of TREWS

“…We share their commitment to rigorous interrogation of the causal assumptions underlying observational clinical studies. In their response 1 to our recent Nature Medicine article 2 , Nemati et al assert that “the findings of Adams et al are likely to be severely biased due to the failure to adjust for ‘processes of care’-related confounding factors.” Nemati et al further claim that “the actual timing of the TREWS alert is of no consequence” in that we might find the same results using an alert that triggered for every patient at a random time. In this response, we first argue that these conclusions are based on strong and unrealistic assumptions about how providers respond to alerts.…”