Clinical predictive models in equine medicine: A systematic review

Abstract

Clinical predictive models use a patient's baseline demographic and clinical data to make predictions about patient outcomes and have the potential to aid clinical decision making. The extent of equine clinical predictive models is unknown in the literature. Using PubMed and Google Scholar, we systematically reviewed the predictive models currently described for use in equine patients. Models were eligible for inclusion if they were published in a peer-reviewed article as a multivariable model used to predict a clinical/laboratory/imaging outcome in an individual horse or herd. The agreement of at least two authors was required for model inclusion. We summarised the patient populations, model development methods, performance metric reporting, validation efforts, and, using the Predictive model Risk of Bias Assessment Tool (PROBAST), assessed the risk of bias and applicability concerns for these models. In addition, we summarised the index conditions for which models were developed and provided detailed information on included models. A total of 90 predictive models and 9 external validation studies were included in the final systematic review. A plurality of models (41%) was developed to predict outcomes associated with colic, for example, need for surgery or survival to discharge. All included models were at high risk of bias, defined as failing one or more PROBAST signalling questions, primarily for analysis-related reasons. Importantly, a high risk of bias does not necessarily mean that models are unusable, but that they require more careful consideration prior to clinical use. Concerns about applicability were low for the majority of models.Systematic reviews such as this can serve to increase veterinarians' awareness of predictive models, including evaluation of their performance and their use in different patient populations.

Letter regarding “Developing a predictive model for spinal shock in dogs with spinal cord injury”

“… 4 Importantly, just because a model is at high ROB does not mean that it has no clinical utility. 2 In a recent systematic review of CPMs in equine medicine, the authors found 90/90 models to be at high ROB, which is consistent with similarly high rates found in systematic reviews of CPMs in the human medical literature. 2 , 5 Clearly, there is an extremely high bar to clear for a CPM to be considered at low risk of bias, and some causes of a high ROB, such as low patient numbers compared to candidate predictor variables, will always be challenging to avoid.…”

“… 2 In a recent systematic review of CPMs in equine medicine, the authors found 90/90 models to be at high ROB, which is consistent with similarly high rates found in systematic reviews of CPMs in the human medical literature. 2 , 5 Clearly, there is an extremely high bar to clear for a CPM to be considered at low risk of bias, and some causes of a high ROB, such as low patient numbers compared to candidate predictor variables, will always be challenging to avoid. Nevertheless, it is important to be open about potential risks of bias when writing about a new CPM so that clinicians are better suited to appraise the model and how much they believe in its predictions.…”

“…Bias is quite common in CPMs and occurs when predicted results systematically deviate from the truth. 2 Risk of bias assessments are increasingly performed for appraisal of CPMs in human medicine using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) in its long or short form. 3 , 4 The population on which the model is developed, the predictors used, and the outcomes assessed can all result in a model being at high ROB.…”

“…To put this into context, an AUC of 0.5 indicates that the model discriminates no better than a coin flip, while an AUC of 0.7‐0.8 is considered to be acceptable, 0.8‐0.9 is considered to be excellent, and greater than 0.9 is outstanding. 2 While we do not yet know how this spinal shock model will perform in an external validation, it is reasonable to assume performance will drop off to some degree.…”

“…Yet, many predictive models fail to see use because there is no simple tool with which to apply the model. 2 , 6 , 7 For the current model, the authors do well to provide the full model equation and even demonstrate the equation's utility with a couple clinical vignettes. 1 Unfortunately, however, they do not provide a simple tool like a sum score or spreadsheet that would allow others to quickly perform model calculations for dogs they are managing.…”