An integrated nomogram combining deep learning, Prostate Imaging–Reporting and Data System (PI-RADS) scoring, and clinical variables for identification of clinically significant prostate cancer on biparametric MRI: a retrospective multicentre study
Abstract:Background Biparametric MRI (comprising T2-weighted MRI and apparent diffusion coefficient maps) is increasingly being used to characterise prostate cancer. Although previous studies have combined Prostate Imaging-Reporting & Data System (PI-RADS)-based MRI findings with routinely available clinical variables and with deep learning-based imaging predictors, respectively, for prostate cancer risk stratification, none have combined all three. We aimed to construct an integrated nomogram (referred to as ClaD) com… Show more
“…12/28 papers deployed fully-automated AI methods based on DL methods and were therefore screened using CLAIM, while 16/28 papers used TML methods to develop semi-automated AI approaches and were assessed using RQS. Of these, 5/12 (42%) DL papers [ 20 – 24 ] and 12/16 (43%) TML papers [ 25 – 36 ] passed the quality screening and were subject to full QUADAS-2 assessment, data extraction, and narrative synthesis. Fig.…”
Section: Resultsmentioning
confidence: 99%
“…None of the papers performed phantom studies to detect scanner-dependent features (Q3), reported calibration statistics (Q10), registered a prospective study (Q11), and reported on the cost-effectiveness of the clinical application of the proposed models (Q14). Only one (8%) paper [ 32 ] discussed a potential biological correlate for some radiomic features included in the final model (Q7), and only two papers [ 28 , 36 ] performed external testing of their models (Q12). Furthermore, only six out of 12 (50%) papers [ 25 , 26 , 29 – 32 ] had image segmentation performed by multiple radiologists or instead assessed the robustness of radiomic features to ROI morphological perturbations (Q2).…”
Section: Resultsmentioning
confidence: 99%
“…Furthermore, only six out of 12 (50%) papers [ 25 , 26 , 29 – 32 ] had image segmentation performed by multiple radiologists or instead assessed the robustness of radiomic features to ROI morphological perturbations (Q2). Eight out of 12 (67%) papers [ 25 – 27 , 30 – 32 , 34 , 35 ] did not make available any images, code, or feature values used to train the models (Q16), and only 4/12 (33%) papers [ 30 , 31 , 34 , 36 ] incorporated non-radiomic features into the multivariable analysis (Q6). The results of RQS screening and full assessment are presented in Additional file 3 .…”
Section: Resultsmentioning
confidence: 99%
“…2 . Overall, 11/17 (65%) [ 12 , 20 – 23 , 25 , 26 , 29 , 31 , 34 , 36 ], 1/17 [ 35 ], and 5/17 [ 24 , 27 , 30 , 32 , 33 ] papers had low, unclear, and high risk of bias, respectively. All papers had low applicability concerns.…”
Section: Resultsmentioning
confidence: 99%
“…Ten out of 12 (83%) TML papers [ 12 , 25 , 26 , 30 – 36 ] utilised non-publicly available institutional datasets, with the remaining 2/12 (17%) studies [ 27 , 29 ] using the PROSTATEx challenge dataset [ 49 ]. In eight (67%) papers [ 25 – 27 , 29 – 32 , 34 ], the histopathological ground truth was obtained using targeted biopsy, while two studies [ 28 , 33 ] relied on radical prostatectomy data, one [ 36 ] was a multi-institutional study relying on either biopsy (targeted or systematic) or prostatectomy data in different cohorts, and one [ 35 ] did not explicitly report the source of ground truth. Only two (17%) papers [ 31 , 32 ] reported the time between biopsy and MRI; in these studies, biopsy was performed either three months [ 31 ] or six weeks [ 32 ] prior to MRI.…”
Objectives
We systematically reviewed the current literature evaluating the ability of fully-automated deep learning (DL) and semi-automated traditional machine learning (TML) MRI-based artificial intelligence (AI) methods to differentiate clinically significant prostate cancer (csPCa) from indolent PCa (iPCa) and benign conditions.
Methods
We performed a computerised bibliographic search of studies indexed in MEDLINE/PubMed, arXiv, medRxiv, and bioRxiv between 1 January 2016 and 31 July 2021. Two reviewers performed the title/abstract and full-text screening. The remaining papers were screened by four reviewers using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) for DL studies and Radiomics Quality Score (RQS) for TML studies. Papers that fulfilled the pre-defined screening requirements underwent full CLAIM/RQS evaluation alongside the risk of bias assessment using QUADAS-2, both conducted by the same four reviewers. Standard measures of discrimination were extracted for the developed predictive models.
Results
17/28 papers (five DL and twelve TML) passed the quality screening and were subject to a full CLAIM/RQS/QUADAS-2 assessment, which revealed a substantial study heterogeneity that precluded us from performing quantitative analysis as part of this review. The mean RQS of TML papers was 11/36, and a total of five papers had a high risk of bias. AUCs of DL and TML papers with low risk of bias ranged between 0.80–0.89 and 0.75–0.88, respectively.
Conclusion
We observed comparable performance of the two classes of AI methods and identified a number of common methodological limitations and biases that future studies will need to address to ensure the generalisability of the developed models.
“…12/28 papers deployed fully-automated AI methods based on DL methods and were therefore screened using CLAIM, while 16/28 papers used TML methods to develop semi-automated AI approaches and were assessed using RQS. Of these, 5/12 (42%) DL papers [ 20 – 24 ] and 12/16 (43%) TML papers [ 25 – 36 ] passed the quality screening and were subject to full QUADAS-2 assessment, data extraction, and narrative synthesis. Fig.…”
Section: Resultsmentioning
confidence: 99%
“…None of the papers performed phantom studies to detect scanner-dependent features (Q3), reported calibration statistics (Q10), registered a prospective study (Q11), and reported on the cost-effectiveness of the clinical application of the proposed models (Q14). Only one (8%) paper [ 32 ] discussed a potential biological correlate for some radiomic features included in the final model (Q7), and only two papers [ 28 , 36 ] performed external testing of their models (Q12). Furthermore, only six out of 12 (50%) papers [ 25 , 26 , 29 – 32 ] had image segmentation performed by multiple radiologists or instead assessed the robustness of radiomic features to ROI morphological perturbations (Q2).…”
Section: Resultsmentioning
confidence: 99%
“…Furthermore, only six out of 12 (50%) papers [ 25 , 26 , 29 – 32 ] had image segmentation performed by multiple radiologists or instead assessed the robustness of radiomic features to ROI morphological perturbations (Q2). Eight out of 12 (67%) papers [ 25 – 27 , 30 – 32 , 34 , 35 ] did not make available any images, code, or feature values used to train the models (Q16), and only 4/12 (33%) papers [ 30 , 31 , 34 , 36 ] incorporated non-radiomic features into the multivariable analysis (Q6). The results of RQS screening and full assessment are presented in Additional file 3 .…”
Section: Resultsmentioning
confidence: 99%
“…2 . Overall, 11/17 (65%) [ 12 , 20 – 23 , 25 , 26 , 29 , 31 , 34 , 36 ], 1/17 [ 35 ], and 5/17 [ 24 , 27 , 30 , 32 , 33 ] papers had low, unclear, and high risk of bias, respectively. All papers had low applicability concerns.…”
Section: Resultsmentioning
confidence: 99%
“…Ten out of 12 (83%) TML papers [ 12 , 25 , 26 , 30 – 36 ] utilised non-publicly available institutional datasets, with the remaining 2/12 (17%) studies [ 27 , 29 ] using the PROSTATEx challenge dataset [ 49 ]. In eight (67%) papers [ 25 – 27 , 29 – 32 , 34 ], the histopathological ground truth was obtained using targeted biopsy, while two studies [ 28 , 33 ] relied on radical prostatectomy data, one [ 36 ] was a multi-institutional study relying on either biopsy (targeted or systematic) or prostatectomy data in different cohorts, and one [ 35 ] did not explicitly report the source of ground truth. Only two (17%) papers [ 31 , 32 ] reported the time between biopsy and MRI; in these studies, biopsy was performed either three months [ 31 ] or six weeks [ 32 ] prior to MRI.…”
Objectives
We systematically reviewed the current literature evaluating the ability of fully-automated deep learning (DL) and semi-automated traditional machine learning (TML) MRI-based artificial intelligence (AI) methods to differentiate clinically significant prostate cancer (csPCa) from indolent PCa (iPCa) and benign conditions.
Methods
We performed a computerised bibliographic search of studies indexed in MEDLINE/PubMed, arXiv, medRxiv, and bioRxiv between 1 January 2016 and 31 July 2021. Two reviewers performed the title/abstract and full-text screening. The remaining papers were screened by four reviewers using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) for DL studies and Radiomics Quality Score (RQS) for TML studies. Papers that fulfilled the pre-defined screening requirements underwent full CLAIM/RQS evaluation alongside the risk of bias assessment using QUADAS-2, both conducted by the same four reviewers. Standard measures of discrimination were extracted for the developed predictive models.
Results
17/28 papers (five DL and twelve TML) passed the quality screening and were subject to a full CLAIM/RQS/QUADAS-2 assessment, which revealed a substantial study heterogeneity that precluded us from performing quantitative analysis as part of this review. The mean RQS of TML papers was 11/36, and a total of five papers had a high risk of bias. AUCs of DL and TML papers with low risk of bias ranged between 0.80–0.89 and 0.75–0.88, respectively.
Conclusion
We observed comparable performance of the two classes of AI methods and identified a number of common methodological limitations and biases that future studies will need to address to ensure the generalisability of the developed models.
Q uality control of multiparametric magnetic resonance imaging (mpMRI) of the prostate has emerged as an important topic following its widespread adoption and the support received from main international guidelines. 1 Several articles have recently discussed the quality standards that should be applied to mpMRI examinations. 2,3 In this issue of JMRI, Cipollari et al 4 bring attention to a new approach to the topic.Using convolutional neural networks (CNN), the authors proposed and tested an automated image quality classification of prostate mpMRI on a cohort of more than 300 patients. The results are quite promising; the reported accuracy for classifying the image quality was 100% for T2-weighted, diffusion-weighted, and contrast-enhanced MR images, and 92.3% for apparent diffusion coefficient (ADC) maps. The potential clinical impact of this model is not yet determined; but a fully automated tool could enhance the quality control evaluation by MRI scanners operators. Lowquality sequences could be quickly detected and immediately repeated, improving workflow efficiency and healthcare delivery.Yet, the study of Cipollari et al 4 has some limitations. The model uses a binary classification to assess image quality, but a wider scale could be more valuable. Scans may have lower than expected quality, but still provide sufficient data to answer the clinical questions that are posed. And, it may be more difficult to achieve similar accuracy if a more complex classification system is utilized. Generalizability may be limited because these CNN were designed and tested on images acquired using one imaging protocol on a single MR scanner. Thus, external validation studies will be necessary to verify whether the model works as well at other institutions.CNNs are deep learning-based tools, in other words, an artificial intelligence (AI) technique. 5 CNNs are, perhaps, the
BackgroundDeep learning for diagnosing clinically significant prostate cancer (csPCa) is feasible but needs further evaluation in patients with prostate‐specific antigen (PSA) levels of 4–10 ng/mL.PurposeTo explore diffusion‐weighted imaging (DWI), alone and in combination with T2‐weighted imaging (T2WI), for deep‐learning‐based models to detect and localize visible csPCa.Study TypeRetrospective.PopulationOne thousand six hundred twenty‐eight patients with systematic and cognitive‐targeted biopsy‐confirmation (1007 csPCa, 621 non‐csPCa) were divided into model development (N = 1428) and hold‐out test (N = 200) datasets.Field Strength/SequenceDWI with diffusion‐weighted single‐shot gradient echo planar imaging sequence and T2WI with T2‐weighted fast spin echo sequence at 3.0‐T and 1.5‐T.AssessmentThe ground truth of csPCa was annotated by two radiologists in consensus. A diffusion model, DWI and apparent diffusion coefficient (ADC) as input, and a biparametric model (DWI, ADC, and T2WI as input) were trained based on U‐Net. Three radiologists provided the PI‐RADS (version 2.1) assessment. The performances were determined at the lesion, location, and the patient level.Statistical TestsThe performance was evaluated using the areas under the ROC curves (AUCs), sensitivity, specificity, and accuracy. A P value <0.05 was considered statistically significant.ResultsThe lesion‐level sensitivities of the diffusion model, the biparametric model, and the PI‐RADS assessment were 89.0%, 85.3%, and 90.8% (P = 0.289–0.754). At the patient level, the diffusion model had significantly higher sensitivity than the biparametric model (96.0% vs. 90.0%), while there was no significant difference in specificity (77.0%. vs. 85.0%, P = 0.096). For location analysis, there were no significant differences in AUCs between the models (sextant‐level, 0.895 vs. 0.893, P = 0.777; zone‐level, 0.931 vs. 0.917, P = 0.282), and both models had significantly higher AUCs than the PI‐RADS assessment (sextant‐level, 0.734; zone‐level, 0.863).Data ConclusionThe diffusion model achieved the best performance in detecting and localizing csPCa in patients with PSA levels of 4–10 ng/mL.Evidence Level3Technical EfficacyStage 2
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
