Advanced cardiovascular risk prediction in the emergency department: updating a clinical prediction model – a large database study protocol

Abstract

Background Patients presenting with chest pain represent a large proportion of attendances to emergency departments. In these patients clinicians often consider the diagnosis of acute myocardial infarction (AMI), the timely recognition and treatment of which is clinically important. Clinical prediction models (CPMs) have been used to enhance early diagnosis of AMI. The Troponin-only Manchester Acute Coronary Syndromes (T-MACS) decision aid is currently in clinical use across Greater Manchester. CPMs have been shown to deteriorate over time through calibration drift. We aim to assess potential calibration drift with T-MACS and compare methods for updating the model. Methods We will use routinely collected electronic data from patients who were treated using TMACS at two large NHS hospitals. This is estimated to include approximately 14,000 patient episodes spanning June 2016 to October 2020. The primary outcome of acute myocardial infarction will be sourced from NHS Digital’s admitted patient care dataset. We will assess the calibration drift of the existing model and the benefit of updating the CPM by model recalibration, model extension and dynamic updating. These models will be validated by bootstrapping and one step ahead prequential testing. We will evaluate predictive performance using calibrations plots and c-statistics. We will also examine the reclassification of predicted probability with the updated TMACS model. Discussion CPMs are widely used in modern medicine, but are vulnerable to deteriorating calibration over time. Ongoing refinement using routinely collected electronic data will inevitably be more efficient than deriving and validating new models. In this analysis we will seek to exemplify methods for updating CPMs to protect the initial investment of time and effort. If successful, the updating methods could be used to continually refine the algorithm used within TMACS, maintaining or even improving predictive performance over time. Trial registration ISRCTN number: ISRCTN41008456

Is your clinical prediction model past its sell by date?

“…We have built a database of more than 13 000 patients using data from multiple hospitals and outcome data from NHS Digital. We will be able to update the algorithm after each patient interaction thereby enabling the simulation of dynamic updating in the hope of countering calibration drift 11. Demographic factors such as ethnicity and measures of deprivation have been shown in the pandemic to have significant prognostic value for disease outcomes; these variables could be incorporated in algorithms 12…”

“…There is an abundance of overlapping CPMs and some would argue that deriving and validating entirely new models is not an efficient use of research funding 15. The focus is shifting to identifying calibration drift and then updating the models 11 16. This emphasises the importance of post-implementation research to verify that CPMs are retaining their calibration and diagnostic accuracy.…”