Validation and Utility Testing of Clinical Prediction Models

Abstract

The growth in the publication of clinical prediction models (CPMs) has been exponential, largely as a result of an ever-increasingavailabilityofclinicaldata,inexpensivecomputational power, and an expanding tool kit for constructing predictive algorithms. Such an abundance of CPMs has led to an overcrowded, confusing landscape in which it is difficult to identify and select the best, most useful models. 1 Few models are externally validated by the same researchers who developed them, and even fewer by independentinvestigators.Only592(43.3%)of1366cardiovascular CPMs in the Tufts PACE Clinical Predictive Model Registry reported at least 1 validation. 2 The proportions of models in the Tufts registry that reported at least 2, 3, and 10 validations were 20.1%, 12.8%, and 2.9%, respectively. 2 A few select CPMs, such as the Framingham Risk Score and EuroSCORE, have had numerous validations.However, even these models are subject to modifications (eg, adding or removing a predictor variable), with the resulting modified model not revalidated externally. Fragmented efforts that assess only one model at a time do not allow for reliable ranking of the comparative performance of the many CPMs available for the same clinical application. A small number of VIEWPOINT

Validated models for pre-test probability of stable coronary artery disease: a systematic review suggesting how to improve validation procedures

“…The copyright holder for this this version posted November 30, 2020. ; https://doi.org/10.1101/2020.11.27.20239301 doi: medRxiv preprint increasing availability of large data-sets, and the highly improved computational power seem to have directed large part of recent researches towards model development rather than model validation. 17 First of all, our review makes an important selection among the many developed models by mainly considering those externally validated. Then, it provides insights into the effects of traditional and emerging risk factors, biomarkers, and comorbidities on the PTP of obstructive CAD.…”

“…For the purposes of generalisation of a PTP model to populations that differ from the development population study, the computation of performance indexes is not sufficient because a lower performance is usually expected. 17,24 Therefore, we also noted whether more extended validation procedures were performed in order to properly apply a model to new populations.…”

“…15,16 To face the flaws on available PTP models highlighted by NICE and ESC, these organizations clearly underline the need for more information on the various risk factors acting as modifier of the PTP, especially in the low probability range, 10 and for the development and validation of new scores addressing outstanding uncertainties in the estimation of the PTP of CAD. 11 This review provides several new contributions to the actual debate on how to ameliorate the PTP models as it focuses on external validation mainly 17 identifies the best results and characterizes the best procedures in terms of significant predictive variables, discriminatory ability and methods completeness. Moreover, the review highlights some key issues that could be further improved in the development and validation phases, to increase decision making capability.…”

L’épidémie de COVID-19 : une autre histoire pourrait être racontée

“…L’accélération donnée par l’épidémie de covid-19 a amplifié l’efflorescence de modèles publiés depuis quelques années. Ils ont l’avantage, pour leurs auteurs, de traiter un grand nombre de données et de présenter des significations statistiques qui font oublier les incertitudes concernant la validité des sélections sur lesquelles ils s’appuient [18] .…”

Multimethod, multidataset analysis reveals paradoxical relationships between sociodemographic factors, Hispanic ethnicity and diabetes

“…For these reasons-and given recent emphasis on the need to perform external validation of analyses of large data sets-we sought to identify sociodemographic factors associated with county-level DM prevalence and to validate our analyses using individual-level DM data. 14 The collective understanding of diabetes and its associated risk factors will likely be furthered by the rapid growth and utilization of machine learning (ML). In fact, ML models, as compared with traditional epidemiological models, have shown promise in studying disease prevalence and explaining variation in health outcomes, including risk factors and complications associated with diabetes.…”