Machine learning algorithm to predict anterior cruciate ligament revision demonstrates external validity

Abstract

Purpose External validation of machine learning predictive models is achieved through evaluation of model performance on different groups of patients than were used for algorithm development. This important step is uncommonly performed, inhibiting clinical translation of newly developed models. Machine learning analysis of the Norwegian Knee Ligament Register (NKLR) recently led to the development of a tool capable of estimating the risk of anterior cruciate ligament (ACL) revision (https://swastvedt.shinyapps.io/calculator_rev/). The purpose of this study was to determine the external validity of the NKLR model by assessing algorithm performance when applied to patients from the Danish Knee Ligament Registry (DKLR). Methods The primary outcome measure of the NKLR model was probability of revision ACL reconstruction within 1, 2, and/or 5 years. For external validation, all DKLR patients with complete data for the five variables required for NKLR prediction were included. The five variables included graft choice, femur fixation device, KOOS QOL score at surgery, years from injury to surgery, and age at surgery. Predicted revision probabilities were calculated for all DKLR patients. The model performance was assessed using the same metrics as the NKLR study: concordance and calibration. Results In total, 10,922 DKLR patients were included for analysis. Average follow-up time or time-to-revision was 8.4 (± 4.3) years and overall revision rate was 6.9%. Surgical technique trends (i.e., graft choice and fixation devices) and injury characteristics (i.e., concomitant meniscus and cartilage pathology) were dissimilar between registries. The model produced similar concordance when applied to the DKLR population compared to the original NKLR test data (DKLR: 0.68; NKLR: 0.68–0.69). Calibration was poorer for the DKLR population at one and five years post primary surgery but similar to the NKLR at two years. Conclusion The NKLR machine learning algorithm demonstrated similar performance when applied to patients from the DKLR, suggesting that it is valid for application outside of the initial patient population. This represents the first machine learning model for predicting revision ACL reconstruction that has been externally validated. Clinicians can use this in-clinic calculator to estimate revision risk at a patient specific level when discussing outcome expectations pre-operatively. While encouraging, it should be noted that the performance of the model on patients undergoing ACL reconstruction outside of Scandinavia remains unknown. Level of evidence III.

Predicting the Objective and Subjective Clinical Outcomes of Anterior Cruciate Ligament Reconstruction: A Machine Learning Analysis of 432 Patients

“…Several studies have revealed the performance of machine learning algorithms in predicting PROMs and the risk of reoperation. 10,17,22,29,30 Ramkumar et al 29,30 applied 7 machine learning models to predict whether patients would achieve the MCIDs of PROMs at 2 years after osteochondral allograft transplant, and the AUCs of the best-performing models ranged from 0.60 to 0.94. Haeberle et al 10 reported AUCs ranging from 0.62 to 0.80 for random forest models predicting the risk of subsequent hip surgeries before primary hip arthroscopy.…”

“…According to Kunze et al, 17 a history of contralateral knee surgery, preoperative knee extension, medial collateral ligament examination, femoral fixation, and BMI were the top 5 features for predicting achievement of the IKDC score MCID, whereas the preoperative IKDC score, the top predictor of our study, was less important. Furthermore, Martin et al 22 analyzed a national knee ligament registry and developed a revision risk calculator for ACLR using only 5 predictive variables (age, preoperative Knee injury and Osteoarthritis Outcome Score [KOOS] Quality of Life subscale, graft choice, femoral fixation, and time from injury to surgery) based on the Cox Lasso model, with a concordance of 0.68. The disparate top predictors in our study may be attributed to the inclusion criteria.…”

Scoping Review on ACL Surgery and Registry Data

“…Recent predictor studies [ 40 – 44 ] harnessed statistical methods, including but not limited to multivariable logistic and linear regressions, Kaplan-Meier survival curves, and Cox proportional hazards models to predict the influence of treatment variables on superior or inferior PROMs and assess the risk of revision ACL surgery over time, respectively, while aiming to account for the complex interaction between a large number of factors. Also, a couple of registry-based studies have already harnessed the use of artificial intelligence (AI) and machine learning (ML) technology in order to predict outcomes following treatment of ACL injury [ 45 •, 46 •]. With the emerging role of AI and ML, registry data will likely enable the development of algorithms designed to predict the individualized risk of ACL-R failure and inferior patient outcomes based on treatment variables [ 45 •, 46 •].…”

“…Also, a couple of registry-based studies have already harnessed the use of artificial intelligence (AI) and machine learning (ML) technology in order to predict outcomes following treatment of ACL injury [ 45 •, 46 •]. With the emerging role of AI and ML, registry data will likely enable the development of algorithms designed to predict the individualized risk of ACL-R failure and inferior patient outcomes based on treatment variables [ 45 •, 46 •].…”

“…Future registry studies will likely implement the use of AI and ML algorithms to analyze large volumes of patient data, develop clinical decision-making tools, and assess the patient-specific risk of inferior treatment outcomes. Currently, only a couple of registry-based studies harnessed the use of AI and ML technology [ 45 •, 46 •], but international collaboration and an increasing volume of data will likely lead to an increase in the popularity of this approach. While AI and ML provide attractive novel approaches to study ACL-R outcomes, it is essential to encourage the development of clinically meaningful models, which augment the surgical decision-making process and translate well to everyday clinical use.…”