How do we estimate survival? External validation of a tool for survival estimation in patients with metastatic bone disease—decision analysis and comparison of three international patient populations

Abstract

BackgroundWe recently developed a clinical decision support tool, capable of estimating the likelihood of survival at 3 and 12 months following surgery for patients with operable skeletal metastases. After making it publicly available on www.PATHFx.org, we attempted to externally validate it using independent, international data.MethodsWe collected data from patients treated at 13 Italian orthopaedic oncology referral centers between 2010 and 2013, then applied to PATHFx, which generated a probability of survival at three and 12-months for each patient. We assessed accuracy using the area under the receiver-operating characteristic curve (AUC), clinical utility using Decision Curve Analysis (DCA), and compared the Italian patient data to the training set (United States) and first external validation set (Scandinavia).ResultsThe Italian dataset contained 287 records with at least 12 months follow-up information. The AUCs for the three-month and 12-month estimates was 0.80 and 0.77, respectively. There were missing data, including the surgeon’s estimate of survival that was missing in the majority of records. Physiologically, Italian patients were similar to patients in the training and first validation sets. However notable differences were observed in the proportion of those surviving three and 12-months, suggesting differences in referral patterns and perhaps indications for surgery.ConclusionsPATHFx was successfully validated in an Italian dataset containing missing data. This study demonstrates its broad applicability to European patients, even in centers with differing treatment philosophies from those previously studied.

Can We Estimate Short- and Intermediate-term Survival in Patients Undergoing Surgery for Metastatic Bone Disease?

“…On external validation, discriminatory ability of the 1-and 6-month models was similar to those previously studied [6,11]. Specifically, the 3-month and 12-month models contained in PATHFx showed AUCs of 0.79 and 0.76, respectively, when confronted with this external validation data set and DCA showed that both of these models were suitable for use in the Scandinavian patient population [6].…”

“…If surgeons are unsure regarding whether an estimate is appropriate, he or she should select ''unknown.'' Doing so will maintain accuracy of the model, while not introducing undue bias, as shown in two prior external validation studies [6,11].…”

“…have been externally validated twice [6,11] and shown to possess clinical utility in other patient populations. Next, the data in the training and validation sets were derived from tertiary referral centers.…”

“…One application designed for use in patients with metastatic bone disease is PATHFx [11]. Introduced in 2011 [4], this validated clinical decision-support tool [6] uses clinical and physiologic variables to generate the probability of survival at 3 and 12 months after orthopaedic surgery.…”

“…In addition, Bayesian networks are capable of estimating the likelihood of rare events, which may be useful when attempting to estimate very short (\ 1 month) life expectancy in a population of surgical candidates. The original Bayesian Belief Networks that we used to estimate 3-and 12-month survival have been externally validated twice [6,11] and possess favorable characteristics on decision curve analysis (DCA) [14], indicating that the models are suitable for clinical use. As such, it is reasonable to determine whether 1-month and 6-month survival could be estimated using similar techniques.…”