A Fundamental Study Assessing the Diagnostic Performance of Deep Learning for a Brain Metastasis Detection Task

Abstract

Increased use of deep convolutional neural networks (DCNNs) in medical imaging diagnosis requires determinate evaluation of diagnostic performance. We performed the fundamental investigation of diagnostic performance of DCNNs using the detection task of brain metastasis. Methods: We retrospectively investigated AlexNet and GoogLeNet using 3117 positive and 37961 negative MRI images with and without metastasis regarding (1) diagnostic biases, (2) the optimal K number of K-fold cross validations (K-CVs), (3) the optimal positive versus negative image ratio, (4) the accuracy improvement curves, (5) the accuracy range prediction by the bootstrap method, and (6) metastatic lesion detection by regions with CNNs (R-CNNs). Results: Respectively, AlexNet and GoogLeNet had (1) 50 ± 4.6% and 50 ± 4.9% of the maximal mean ± 95% confidence intervals (95% CIs) measured with equal-sized negative versus negative image datasets and positive versus positive image datasets, (2) no less than 10 and 4 of K number in K-CVs fell within the respective maximum biases of 4.6% or 4.9%, (3) 74% of the highest accuracy with equal positive versus negative image ratio dataset and 91% of that with four times of negative-to-positive image ratio dataset, (4) the accuracy improvement curves increasing from 69% to 74% and 73% to 88% as positive versus negative pairs of the training images increased from 500 to 2495, (5) at least nine and six out of 10-CV result sets essential to predict the accuracy ranges by the bootstrap method, and (6) 50% and 45% of metastatic lesion detection accuracies by R-CNNs. Conclusions: Our research presented methodological fundamentals to evaluate diagnostic features in the visual recognition of DCNNs. Our series will help to conduct the accuracy investigation of computer diagnosis in medical imaging.

Brain metastasis detection using machine learning: a systematic review and meta-analysis

“…After full-text reviews of the 21 provisionally eligible articles, nine articles were excluded because they were not in the field of interest, 31–37 contained potentially overlapping data, 38 or contained insufficient information in terms of detectability of machine learning using MRI data for patients with BM. 39 Finally, 12 articles were included in the present systematic review and meta-analysis. 9–20 …”

Fully Automated MR Detection and Segmentation of Brain Metastases in Non‐small Cell Lung Cancer Using Deep Learning

“…Automated detection and segmentation of BMs in NSCLC pose the following challenges: (I) multifocality, (II) heterogeneity of BMs in terms of size and appearance due to the underlying mutation of the primary tumor, its stage, previously administered treatments, and (III) inhomogeneous imaging data. 7,15,19,26 In this context, the majority of recent studies investigating deep learning-based detection of BMs used homogenous imaging data, mostly a standardized protocol consisting of a distinct 3D T 1 CE sequence at a single institution for planning of radiosurgery, 25,[29][30][31] which limits their generalizability and questions the usefulness in clinical routine. In contrast, the DLM applied in the present study provided a high detection sensitivity on heterogeneous "reallife" imaging data acquired on scanners from different vendors, generations, and study centers with resulting divergent scan parameters and unstandardized application of contrast media.…”

A fundamental study assessing the generalized fitting method in conjunction with every possible coalition of N-combinations (G-EPOC) using the appendicitis detection task of computed tomography

“…However, there are some issues relating to the undefined optimal K value of K-CV. Although K-values are arbitrary, 5 or 10 for K (5- or 10-CV) is recommended in general [ 14 - 17 ]. The most common reason for recommending 5-CV is that it takes less time to calculate than 10-CV.…”

“…In addition, the Pareto principle, otherwise known as the 80/20 law, in which most of the representative values of the whole group is yielded by 20% of the group, might also be related to the recommendation of 5-CV [ 16 , 17 ]. However, 5-CV has a lower accuracy value compared to 10-CV because the accuracy value of DL substantially depends on the training data volume, which is 4/5 of the total data in 5-CV and 9/10 in 10-CV [ 14 ].…”

“…Noguchi et al . proposed a method with which the accuracy value of 10-CV can be estimated using less than 10 results of the 10-CV by the bootstrap method in conjunction with every possible coalition of N-combinations [ 14 ]. They demonstrated that their method estimated the ranges including the mean accuracy value of 10-CV with the use of only 6 of 10 results of the 10-CV at a rate of over 95%; i.e., they showed that their boot-EPOC method could skip 4 of the 10 processing repetitions in 10-CV.…”