Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review

Abstract

Background and purpose — External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines. Material and methods — We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting. Results — We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43–89), with 6 items being reported in less than 4/18 of the studies. Interpretation — Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

Predictive Algorithms in the Diagnosis and Management of Pediatric Hip and Periarticular Infection

“…Decision curve analysis would be extremely valuable in the setting of pediatric BJI, as the net clinical benefit of some interventions remains undefined. A recent systematic review of orthopaedic external validation studies showed that few studies adequately describe these model characteristics 51 . As shown in Table III, the temptation when performing validation is to repeat a regression analysis on a new population without considering these key aspects of algorithm behavior.…”

Can a Bayesian belief network for survival prediction in patients with extremity metastases (PATHFx) be externally validated in an Asian cohort of 356 surgically treated patients?

“…A slight decline in MC was observed when the 6-month and 12-month predictions were compared with the 12-month (MC = 0.86) and 18-month predictions (MC = 0.88), respectively. Since inconsistencies could happen with any survival prediction models, it is important to interpret their estimations onstrated that a machine-learning model's performance could vary in ethnogeographically distinct populations (8,16) and repeated validation in different cohorts is needed (9,15).…”

“…It has also been recently updated to the 3rd version (3) to provide predictions for patients treated both operatively and with radiation only. However, some studies suggested PSSs could perform differently between ethnogeographically distinct cohorts (4,5,(8)(9)(10)(11)(12), and they should have been validated before being applied onto a specific population. The PATHFx offers survival predictions at 6 different time points: 1, 3, 6, 12, 18, and 24 months.…”

International Validation of the SORG Machine-learning Algorithm for Predicting the Survival of Patients with Extremity Metastases Undergoing Surgical Treatment

“…This international external validation study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines The sodium level was missing for 0.03% (1 of 356 patient) of patients in validation cohort and 18% (199 of 1090) of patients in developmental cohort; ALP = alkaline phosphatase. [11,15,16] and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) [12] guidelines. The study was approved by our institutional review board (201912022RIND).…”