Leveraging implicit expert knowledge for non-circular machine learning in sepsis prediction

Abstract

Sepsis is the leading cause of death in non-coronary intensive care units. Moreover, a delay of antibiotic treatment of patients with severe sepsis by only few hours is associated with increased mortality. This insight makes accurate models for early prediction of sepsis a key task in machine learning for healthcare. Previous approaches have achieved high AUROC by learning from electronic health records where sepsis labels were defined automatically following established clinical criteria. We argue that the practice of incorporating the clinical criteria that are used to automatically define ground truth sepsis labels as features of severity scoring models is inherently circular and compromises the validity of the proposed approaches. We propose to create an independent ground truth for sepsis research by exploiting implicit knowledge of clinical practitioners via an electronic questionnaire which records attending physicians' daily judgements of patients' sepsis status. We show that despite its small size, our dataset allows to achieve state-of-the-art AUROC scores. An inspection of learned weights for standardized features of the linear model lets us infer potentially surprising feature contributions and allows to interpret seemingly counterintuitive findings.

Early Prediction of Sepsis in the ICU using Machine Learning: A Systematic Review

“…The results of the literature search, including the numbers of studies screened, assessments for eligibility, and articles reviewed (with reasons for exclusions at each stage), are presented in Figure 1. Out of 974 studies, 22 studies met the inclusion criteria [Abromavičius et al, 2020, Barton et al, 2019, Bloch et al 2019, Calvert et al, 2016, Desautels et al, 2016, Futoma et al, 2017b, Kaji et al, 2019, Kam and Kim, 2017, Lauritsen et al, 2020, Lukaszewski et al, 2008, Mao et al, 2018, McCoy and Das, 2017, Moor et al, 2019, Nemati et al 2018, Reyna et al, 2019, Schamoni et al, 2019, Scherpf et al, 2019, Shashikumar et al, 2017a,b, Sheetrit et al, 2019, Van Wyk et al, 2019, van Wyk et al, 2019]. The majority of excluded studies ( n = 952) did not meet one or multiple inclusion criteria, such as studying a non-human (e.g.…”

“…the employed time window lengths have been varied) [Abromavičius et al, 2020, Nemati et al, 2018]. In one study [Schamoni et al, 2019], sepsis labels were assigned by trained ICU experts. Depending on the definition of sepsis used, and whether subsampling of controls was used to achieve a more balanced class ratio (facilitating the training of machine learning models), the prevalence of patients developing sepsis ranged between 6.2% and 63.6% (Figure 2).…”

“…While initial studies employing machine learning for the prediction of sepsis have demonstrated promising results [Calvert et al, 2016, Desautels et al, 2016, Kam and Kim, 2017], the literature since has been diverging on which are the most pressing open challenges that need to be addressed to further the goal of early sepsis detection. On the one hand, corporations have been propelling the deployment of the first interventional studies [Burdick et al, 2020, Shimabukuro et al, 2017], while on the other hand, recent findings have cast doubt on the validity and meaningfulness of the experimental pipeline that is currently being implemented in most retrospective analyses [Schamoni et al, 2019]. This can be partially attributed to circular prediction settings (for more details, please refer to Section 4.4).…”

“…While this strategy may reduce the problem of circularity, a coherent and comprehensive definition of sepsis cannot be easily guaranteed. Notably, Schamoni et al [2019] report very high inter-rater agreement. They assign, however, only daily labels which is in contrast to automated Sepsis-3 labels that are typically extracted in an hourly resolution.…”

“…Furthermore, it is plausible that even with clinical experts in the loop, some level of (indirect) circularity could still take place, because a clinician would also consult the patients' vital and laboratory measurements in order to assign the sepsis tag, it would merely be less explicit. Since Schamoni et al [2019] proposed a way to circumvent the issue of circularity, this also means that no existing work has empirically assessed the existence (or the relevance) of circularity in machine learning-based sepsis prediction. For Sepsis-3, if the standard 72 h window is used for assessing an increase in SOFA (sequential organ failure assessment score) score, i.e.…”