Joint Weakly and Semi-Supervised Deep Learning for Localization and Classification of Masses in Breast Ultrasound Images

Abstract

We propose a framework for localization and classification of masses in breast ultrasound (BUS) images. We have experimentally found that training convolutional neural network based mass detectors with large, weakly annotated datasets presents a non-trivial problem, while overfitting may occur with those trained with small, strongly annotated datasets. To overcome these problems, we use a weakly annotated dataset together with a smaller strongly annotated dataset in a hybrid manner. We propose a systematic weakly and semi-supervised training scenario with appropriate training loss selection. Experimental results show that the proposed method can successfully localize and classify masses with less annotation effort. The results trained with only 10 strongly annotated images along with weakly annotated images were comparable to results trained from 800 strongly annotated images, with the 95% confidence interval of difference -3.00%-5.00%, in terms of the correct localization (CorLoc) measure, which is the ratio of images with intersection over union with ground truth higher than 0.5. With the same number of strongly annotated images, additional weakly annotated images can be incorporated to give a 4.5% point increase in CorLoc, from 80.00% to 84.50% (with 95% confidence intervals 76.00%-83.75% and 81.00%-88.00%). The effects of different algorithmic details and varied amount of data are presented through ablative analysis.Index Terms-Breast ultrasound, convolutional neural networks, mass classification, mass localization, semisupervised learning, weakly supervised learning.

Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis

“…MIL category Method Brain Tong et al (2014) AD classification global excl bag Chen et al (2015b) cerebral small vessel disease detection global instance Dubost et al (2017) enlarged perivascular space detection local instance Eye Venkatesan et al (2015) diabetic retinopathy classification global excl bag Quellec et al (2012) diabetic retinopathy classification global, local instance Schlegl et al (2015) fluid segmentation local instance Manivannan et al (2016) retinal nerve fiber layer visibility classification global, local instance Lu et al (2017) fluid detection global instance Breast Maken et al (2014) breast cancer detection global multiple Sanchez de la Rosa et al (2015) breast cancer detection global, local excl bag Shin et al (2017) mass localization, classification global, local instance Lung Dundar et al (2007) pulmonary embolism detection false positive instance Bi and Liang (2007) pulmonary embolism detection false positive instance Liang and Bi (2007) pulmonary embolism detection false positive instance Cheplygina et al (2014) COPD classification global multiple Melendez et al (2014) tuberculosis detection global, local instance Stainvas et al (2014) lung cancer lesion classification false positive instance Melendez et al (2016) tuberculosis detection global, local instance Kim and Hwang (2016) tuberculosis detection global, local instance Shen et al (2016) lung cancer malignancy prediction global, local instance Cheplygina et al (2017) COPD classification global instance Li et al (2017b) abnormality detection (14 classes) global, local instance Abdomen Dundar et al (2007) polyp detection false positive instance Wu et al (2009) polyp detection false positive instance Lu et al (2011) polyp detection, size estimation false positive instance Wang et al (2012) polyp detection false positive instance Wang et al (2015a) lesion detection global prim bag Wang et al (2015b) lesion detection global prim bag Histology/Microscopy Dundar et al (2010) brea...…”

“…The results show that, when instance classification is the goal, adding more labeled bags does not necessarily increase instancelevel performance. Shin et al (2017) use both bag and instance labels for localization and classification of breast masses. They show that bag labels should be given less weight than the instance labels -i.e.…”

Deep Learning-Based Breast Cancer Diagnosis at Ultrasound: Initial Application of Weakly-Supervised Algorithm Without Image Annotation Original Research

“…Automated determination of regions has been proposed using various methods of region proposals. Diagnostic performances using the automatically determined regions were suboptimal or metrics were not-well demonstrated with the highest accuracy of 87.5% in a study using DenseNet 13 , 60.6% of sensitivity for malignant lesions in a study using fully convolutional networks 23 , no overall diagnostic metrics in a study using faster R-CNN 24 . Image-based classification with weaklysupervised DL algorithms has been proposed in the present study and our previous work 25 .…”

Semi-Supervised Breast Histological Image Classification by Node-Attention Graph Transfer Network

“…There is a major concern that existing semi-supervised medical image classification methods ignore the interactive influence between image samples, which can be addressed by the Graph Convolutional Network (GCN) with a graph learning module [12]. There lefts a major challenge for the automatic histological image classification that is the limited amount of data available under supervised learning and temporally annotating a large number of breast histological images is unsubstantial in clinical application as demonstrated in [1], [18], [30]. Nevertheless, transfer learning provides a novel ideology that has been proved to be effective for solving the challenge of limited labeled histology data [27], [29], [33], and it needs a completely labeled dataset as the source domain with a target domain of partially labeled data.…”