External Validation of the Bone Metastases Ensemble Trees for Survival (BMETS) Machine Learning Model to Predict Survival in Patients With Symptomatic Bone Metastases

Abstract

PURPOSE The Bone Metastases Ensemble Trees for Survival (BMETS) model uses a machine learning algorithm to estimate survival time following consultation for palliative radiation therapy for symptomatic bone metastases (SBM). BMETS was developed at a tertiary-care, academic medical center, but its validity and stability when applied to external data sets are unknown. PATIENTS AND METHODS Patients treated with palliative radiation therapy for SBM from May 2013 to May 2016 at two hospital-based community radiation oncology clinics were included, and medical records were retrospectively reviewed to collect model covariates and survival time. The Kaplan-Meier method was used to estimate overall survival from consultation to death or last follow-up. Model discrimination was estimated using time-dependent area under the curve (tAUC), which was calculated using survival predictions from BMETS based on the initial training data set. RESULTS A total of 216 sites of SBM were treated in 182 patients. Most common histologies were breast (27%), lung (23%), and prostate (23%). Compared with the BMETS training set, the external validation population was older (mean age, 67 v 62 years; P < .001), had more primary breast (27% v 19%; P = .03) and prostate cancer (20% v 12%; P = .01), and survived longer (median, 10.7 v 6.4 months). When the BMETS model was applied to the external data set, tAUC values at 3, 6, and 12 months were 0.82, 0.77, and 0.77, respectively. When refit with data from the combined training and external validation sets, tAUC remained > 0.79. CONCLUSION BMETS maintained high discriminative ability when applied to an external validation set and when refit with new data, supporting its generalizability, stability, and the feasibility of dynamic modeling.

Improving providers' survival estimates and selection of prognosis‐ and guidelines‐appropriate treatment for patients with symptomatic bone metastases: Development of the Bone Metastases Ensemble Trees for Survival Decision Support Platform

“…This is substantially better than reported by comparable survival models used in this context, where calibration at similar survival times may vary by ± 20%–30% or more 28 . Two external validation studies for BMETS have reported excellent discrimination and acceptable calibration over the prediction range of the model 29,30 …”

“…28 Two external validation studies for BMETS have reported excellent discrimination and acceptable calibration over the prediction range of the model. 29,30 To promote selection of palliative RT regimens more closely aligned with predicted patient prognosis and best evidence in the management of symptomatic bone metastases, we sought to develop the providerfacing BMETS Bone Metastases Ensemble Trees for Survival-Decision Support Platform (BMETS-DSP) for use in this patient population. We aimed to design a tool that (1) collects patient-specific demographic,…”

Assessment of tissue toxicity risk in breast radiotherapy using Bayesian networks

“…Artificial intelligence (AI) and machine learning (ML) are well suited to be applied to radiotherapy for decision support tools that improve radiotherapy treatment options, 6–9 aid in toxicity and treatment response prediction, 10,11 and provide patient‐specific dosimetric decision making 12 . AI and ML models can identify key features and causal relationships where conventional analysis cannot.…”

“…1,2 The ACCEL trial is a recent phase II prospective cohort clinical trial, 3 which aims to assess the toxicity of an APBI treatment regimen with unique treatment planning and delivery technique over only five daily treatments. 4,5 Artificial intelligence (AI) and machine learning (ML) are well suited to be applied to radiotherapy for decision support tools that improve radiotherapy treatment options, [6][7][8][9] aid in toxicity and treatment response prediction, 10,11 and provide patient-specific dosimetric decision making. 12 AI and ML models can identify key features and causal relationships where conventional analysis cannot.…”

Evaluation of the Clinical Utility of the Bone Metastases Ensemble Trees for Survival Decision Support Platform (BMETS-DSP): A Case-Based Pilot Assessment

“…[1][2][3][4] Moreover, available evidence-based and consensus guidelines do not provide clear criteria for selecting between the range of palliative RT regimens. [5][6][7] To address these issues, we developed the providerfacing Bone Metastases Ensemble Trees for Survival Decision Support Platform (BMETS-DSP), 8 which (1) collects patient-specific characteristics critical to treatment selection, (2) displays a patient-specific predicted survival curve on the basis of the validated BMETS machine learning survival model, [9][10][11][12] and (3) provides case-specific, evidence-based recommendations for RT, open surgery, systemic therapy, and hospice referral in the care of symptomatic bone metastases. 8 The BMETS-DSP is available on a free-access website.…”