Parallel Structure Deep Neural Network Using CNN and RNN with an Attention Mechanism for Breast Cancer Histology Image Classification

Abstract

In this paper, we present a new deep learning model to classify hematoxylin–eosin-stained breast biopsy images into four classes (normal tissues, benign lesions, in situ carcinomas, and invasive carcinomas). Our model uses a parallel structure consist of a convolutional neural network (CNN) and a recurrent neural network (RNN) for image feature extraction, which is greatly different from the common existed serial method of extracting image features by CNN and then inputting them into RNN. Then, we introduce a special perceptron attention mechanism, which is derived from the natural language processing (NLP) field, to unify the features extracted by the two different neural network structures of the model. In the convolution layer, general batch normalization is replaced by the new switchable normalization method. And the latest regularization technology, targeted dropout, is used to substitute for the general dropout in the last three fully connected layers of the model. In the testing phase, we use the model fusion method and test time augmentation technology on three different datasets of hematoxylin–eosin-stained breast biopsy images. The results demonstrate that our model significantly outperforms state-of-the-art methods.

Breast Cancer Histopathology Image Classification Using an Ensemble of Deep Learning Models

“…In this way, 675 images were used for training whereas the remaining 170 images were kept for testing the model. Following [35], we used 5-fold cross-validation on training images which means that 540 images were used for training and 135 images for validation purpose. Again, we have an equal percentage of non-carcinoma and carcinoma images in training and validation.…”

Experimental Assessment of Color Deconvolution and Color Normalization for Automated Classification of Histology Images Stained with Hematoxylin and Eosin

“…This consists of a range of activities such as the acquisition, storage, sharing, analysis and interpretation of histological images [ 10 ]. In this domain, computer-assisted classification of tissue samples has attracted considerable research interest in recent years as a means for assisting pathologists in several tasks, for instance, the classification of specimens into normal or abnormal [ 11 , 12 , 13 , 14 ], the grading of neoplastic tissue [ 15 , 16 , 17 , 18 ], the estimation of tumor proliferation [ 19 ] and the identification of tissue substructures such as epithelium, stroma, lymphocytes, necrosis, etc. [ 20 , 21 ].…”

The use of artificial intelligence, machine learning and deep learning in oncologic histopathology

“…Most recently, RNNs have shown excellent results in the classification of histopathological images. Yao et al 21 demonstrated superior results with a parallel system of CNN and RNN in the image classification of four classes of breast biopsy histopathology images (normal breast tissue, benign, carcinoma in situ, and invasive carcinoma) compared to CNN alone. Iizuka et al 22 used a combined CNN and RNN on WSIs of stomach and colon biopsies.…”