Machine learning in renal cell carcinoma research: the promise and pitfalls of ‘renal‐izing’ the potential of artificial intelligence

Abstract

Artificial intelligence and machine learning (ML) are increasingly applied to the study of patients with RCC [1]. Advanced techniques, such as neural networks or random forests, can be used to analyse a vast amount of clinical data to uncover specific prognostic features that may not be detectable with traditional statistical methods. However, despite some promising results, recent studies show that the benefits of ML are not ubiquitous, especially when deployed on imperfect and non-granular datasets [2].In this issue of the BJUI, Boulenger de Hauteclocque employ data from the French multi-institutional kidney cancer database UroCCR (ClinicalTrials.gov Identifier: NCT03293563), to investigate the ability of seven ML algorithms to predict pT3a upstaging in a cohort of patients who underwent surgery for cT1/cT2a RCC. Using supervised ML algorithms, they reported a prediction accuracy (measured by the area under the receiver-operating curve) of 0.77 for their best model [3]. While the results are intriguing, the design of the study raises questions that should be considered by clinicians when interpreting the results.

Reply to: machine learning in renal cell carcinoma research: promise and pitfalls of ‘renal‐izing’ the potential of artificial intelligence

“…

We wish to thank Khene et al [1] for their comments on our article [2], thus offering us the opportunity to clarify important methodological points.

…”

“…The training and testing datasets were similar according to both patients' characteristics and upstaging rate (15.2% in each of them; see Table S2). Khene et al [1] challenged the choice of the training and testing datasets, by mentioning that the database had a high rate of missing observations, notably for R.E.N.A.L. (Radius, Exophytic/Endophytic, Nearness, Anterior/Posterior, Location) nephrometry score.…”

“…The authors [1] found the estimated receiver operating characteristic area under the curve of 0.77 on the test dataset 'intriguing'. They omitted its estimated value of 0.78 on the training dataset using 10-fold cross-validation (Fig.…”

“…All the results ensured good discrimination and calibration abilities of the predictive signature, as well as its generalisation on an external validation dataset. Following the authors [1] comments about clinical utility, we have provided the decision curve (Fig. 1), showing higher net reduction in interventions per 100 patients (depending on tumour size, active surveillance or ablation could be considered as valid treatment options in that setting), by using the model for decision making across the range of relevant threshold probabilities as compared to treating all or treating none [3].…”

The BJUI Editorial Team's view on artificial intelligence and machine learning

“…The best prediction model achieved an area under the receiver-operating characteristic curve of 0.77. Khene et al [2] on behalf of the European Association of Urology-Young Academic Urologists (EAU-YAU) Renal Cancer Working Group in their letter to the Editor raise the very relevant issues of: the problem of handling missing data and imputing approaches, adjustable hyperparameters, differentially weighting input values, methods used to evaluate the predictive accuracy of the model, and questioning the clinical relevance of such a model. AI prediction models have made an amazingly rapid introduction and widespread use into clinical management [3] with often insufficient validation, e.g., the Epic Sepsis Model (ESM) widely implemented in United States hospitals and poorly predicting the onset of sepsis [4].…”