Evaluating performance of early warning indices to predict physiological instabilities

Abstract

Patient monitoring algorithms that analyze multiple features from physiological signals can produce an index that serves as a predictive or prognostic measure for a specific critical health event or physiological instability. Classical detection metrics such as sensitivity and positive predictive value are often used to evaluate new patient monitoring indices for such purposes, but since these metrics do not take into account the continuous nature of monitoring, the assessment of a warning system to notify a user of a critical health event remains incomplete. In this article, we present challenges of assessing the performance of new warning indices and propose a framework that provides a more complete characterization of warning index performance predicting a critical event that includes the timeliness of the warning. The framework considers 1) an assessment of the sensitivity to provide a notification within a meaningful time window, 2) the cumulative sensitivity leading up to an event, 3) characteristics on if the warning stays on until the event occurs once a warning has been activated, and 4) the distribution of warning times and the burden of additional warnings (e.g., false-alarm rate) throughout monitoring that may or may not be associated with the event of interest. Using an example from an experimental study of hemorrhage, we examine how this characterization can differentiate two warning systems in terms of timeliness of warnings and warning burden.

A framework to characterize the performance of early warning index alarm systems for patient monitoring

“…In addition, scenarios where multiple warnings prior to an event occur make calculating traditional performance indices such as false positive rate and sensitivity not straight forward and intuitive. In presence of such scenarios, researchers may only count the warning closest to an event disregarding the remainder of warnings on the record [7] or sometimes look at the performance metric as a function of time [8].…”

“…We propose an approach which estimates the above three characteristics of a warning system considering multiple warnings per record, warning timeliness and warning burden (occurrence of multiple warning per event/record) [8]. The method involves identifying the time before an event of interest when warnings are meaningful and evaluating how frequently warnings occur before, within, and after those times.…”

“…We previously presented an example of using the proposed framework for event records in [8] using an experimental animal study of hemorrhage. Here, we extend that presentation by also including non-event records.…”

“…Experiments were performed at the University of Texas Medical Branch under a protocol reviewed and approved by the Institutional Animal Care and Use Committee (IACUC) and conducted in compliance with the guidelines of the National Institutes of Health and the American Physiological Society for the care and use of laboratory animals. For the first experiment, as described in detail in [9] and summarized in [8], 8 female sheep underwent two hemorrhages on different days (one day at 1.25 mL/kg BW /min and the other at 0.25 mL/kg BW /min). The hemorrhage was performed until a hypotensive event occurred that was defined as a drop in mean arterial pressure (MAP) of 30 mmHg.…”

“…Continuous physiological monitoring including arterial blood pressure was performed and recorded through the baseline (at least 30 min), hemorrhage, and post-hemorrhage periods. The arterial pressure recordings were analyzed for this study as described in [8].…”