A Multi-Center, Multi-Vendor Study to Evaluate the Generalizability of a Radiomics Model for Classifying Prostate cancer: High Grade vs. Low Grade

Abstract

Radiomics applied in MRI has shown promising results in classifying prostate cancer lesions. However, many papers describe single-center studies without external validation. The issues of using radiomics models on unseen data have not yet been sufficiently addressed. The aim of this study is to evaluate the generalizability of radiomics models for prostate cancer classification and to compare the performance of these models to the performance of radiologists. Multiparametric MRI, photographs and histology of radical prostatectomy specimens, and pathology reports of 107 patients were obtained from three healthcare centers in the Netherlands. By spatially correlating the MRI with histology, 204 lesions were identified. For each lesion, radiomics features were extracted from the MRI data. Radiomics models for discriminating high-grade (Gleason score ≥ 7) versus low-grade lesions were automatically generated using open-source machine learning software. The performance was tested both in a single-center setting through cross-validation and in a multi-center setting using the two unseen datasets as external validation. For comparison with clinical practice, a multi-center classifier was tested and compared with the Prostate Imaging Reporting and Data System version 2 (PIRADS v2) scoring performed by two expert radiologists. The three single-center models obtained a mean AUC of 0.75, which decreased to 0.54 when the model was applied to the external data, the radiologists obtained a mean AUC of 0.46. In the multi-center setting, the radiomics model obtained a mean AUC of 0.75 while the radiologists obtained a mean AUC of 0.47 on the same subset. While radiomics models have a decent performance when tested on data from the same center(s), they may show a significant drop in performance when applied to external data. On a multi-center dataset our radiomics model outperformed the radiologists, and thus, may represent a more accurate alternative for malignancy prediction.

Prostate Cancer Radiogenomics—From Imaging to Molecular Characterization

“…The development of different models in an automatic fashion, using ML and AI techniques, and the construction of nomograms [ 91 ] could further improve the radiomic potential on this issue. The currently existing data are promising, with radiomics outperforming PIRADS v2 in the detection of high-grade versus low-grade PCa, although some limitations remain regarding the standardization of data, and further studies are required to confirm the performance of radiomics compared to conventional radiological analysis [ 92 ]. Moreover, radiomics models are useful in the detection of prostate extracapsular extension (ECE), and allows predictive models to be build for the pretreatment detection of ECE, focusing on a combined model of clinical, conventional radiology and radiomics [ 93 , 94 , 95 ].…”

Single-center versus multi-center biparametric MRI radiomics approach for clinically significant peripheral zone prostate cancer

“…With proper inclusion of these steps, it is expected that both the single-center and multi-center validation performance of the new multi-center model will improve [ 9 , 24 ]. However, proper implementation requires extensive optimization experiments since just selecting a plausible set of post-processing settings does not necessarily lead to an improvement [ 25 ]. Additionally, when looking at the current generalization of the multi-center model it can be speculated that even when omitting these pre-processing steps the multi-center training is able to learn information about the diverse data.…”

“…However, there is a lack of studies that subsequently investigated the performance of their approach or model on McMv data [ 6 ]. Only one recent study by Castillo et al [ 25 ] investigated the generalizability of a radiomics model for classifying PCa. Although this study differed from the current one and used a suboptimal post-processing approach, they also found a lacking generalizability for single-center models [ 25 ].…”

“…Only one recent study by Castillo et al [ 25 ] investigated the generalizability of a radiomics model for classifying PCa. Although this study differed from the current one and used a suboptimal post-processing approach, they also found a lacking generalizability for single-center models [ 25 ]. The test AUC for their single-center models was lower than ours (0.816 vs. 0.75) and the multi-center model did not manage to improve on this even with preprocessing.…”

“…Even with extensive experimentation, the effects of various preprocessing on improving model generalization remain topic of research. For example, Castillo et al [ 25 ] implemented pre-processing for their radiomics MRI PCa model that did not lead to better performance (AUC single center model 0.75 vs. AUC multi-center model 0.75). All these techniques aim to improve generalization on multicenter data.…”