When to customize a severity model

Abstract

General intensive care unit (ICU) severity models are remarkably popular, as judged by the number of peer review publications that contain a severity system as a key component of the study. A MEDLINE search of articles published between January 1993 and December 1997 revealed that there were 552 articles published, with two-thirds of these studies using the Acute Physiology and Chronic Health Evaluation (APACHE), mortality probability models (MPMs), or the Simplified Acute Physiology Score (SAPS) (S. Weitzen, T. Higgins, D. Teres, unpublished data). Studies using pediatric or neonatal models were next in frequency (15 %), with a much smaller number of articles focused on cardiac surgery models, multiple organ failure models, or trauma scores. In more than half of these studies the severity score was used for risk stratification or as a clinical descriptor of patient populations. Surprisingly, 30 % of the articles focused on the development, validation, or performance of severity systems or comparisons of models. Other studies (10 %) used models as part of economic analyses and in only 4 % of studies was the primary goal comparison of quality of care in ICUs.In this issue of Intensive Care Medicine, Metnitz et al. [1] report, in a small but detailed study, on the performance of SAPS II in nine Austrian ICUs. The results are not surprising; SAPS II did not fit well in the new set of patients 6 years after the original publication of SAPS II [2]. To date, there have been no studies that have shown that severity models are stable over time, in a new setting, and with different case mixes [3±7]. APACHE III did not demonstrate external validation using a large number of hospitals in the United States [8,9]. If we believe in medical progress and in the advancement of science, then we would not expect the models to show good calibration over time. We would expect the observed mortality to be lower than the mean predicted hospital mortality, particularly for middle severity patients. If there were severe resource constraints and reduction in staffing and we had good benchmark data, we might expect to find higher observed than predicted hospital mortality. What about the introduction of a new disease now more frequently treated in the ICU but not previously included in severity model development? AIDS was considered a fatal disease and was only minimally included in APACHE, MPM, and SAPS databases [2, 10±12]. Now AIDS is a more chronic disease with more patients being admitted to the ICU with AIDS as a background condition or as a major component of the acute process. What about new technology? Non-invasive positive pressure ventilation is now being more commonly used. For a patient with acute exacerbation of chronic respiratory failure who is placed on nocturnal nasal ventilation on admission to the ICU, how do we apply the definitions of respiratory failure including intubation and mechanical ventilation as described in models that are now several years old? For all of the above reasons, there is a rationale for consider...

Validation of severity scoring systems SAPS II and APACHE II in a single-center population

“…Therefore the group of patients studied appears to be homogeneous for the ICUs considered, even if collected over a long period. It has been claimed that the severity systems are not expected to show good calibration over time due to medical progress [17]. Nevertheless, overall good calibration has been reported for APACHE II 10 years after publication [16].…”

Hospital mortality is associated with ICU admission time

“…To date, there have been no studies that have shown that prognostic models are stable over time, in a new setting, and with different case mixes [14, 15]. As a result of medical progress and advancement of science, it is expected that the model’s performance will decline over time [16]. To account for this decline in performance we recalibrated the APACHE II before using it for case-mix correction.…”

The state of research on multipurpose severity of illness scoring systems: are we on target?

“…[2,3,19,20]. Now, in this issue of Intensive Care Medicine, Suistomaa et al [21] report, in a small but well-conducted prospective study, the effect of sampling rate of laboratory and haemodynamic data on severity scoring systems.…”

“…At the end of the discussion, they call for ª... further studies ...to measure the exact magnitude of the bias resulting from different sampling methodsº. As others have already pointed out when discussing the pros and cons of customization of severity models, when poor goodnessof-fit is documented [2,20] such an appeal could generate, as an unintended consequence, a flood of studies and papers demonstrating the sensitivity of SISS to methods of data collection and the production of different methods supposed to take into account such phenomena. We do not need these kinds of studies as we already know everything about our SISS.…”

“…After the first publication by Knaus et al [1], in 1981, on the issue of measuring, explaining and predicting intensive care unit (ICU) performance, that established the scientific foundation for the discipline of outcome prediction models in intensive care, hundreds of papers reporting empirical findings and personal positions have been published [2]. Although most of such reports actually deal with the specific issue of conceptualization, validity and appropriateness of the prediction models developed for ICU patients, the definition and measurement of the case-mix and output of an ICU is still a complex, persistent and pervasive problem [3].…”