CNN-RNN: a large-scale hierarchical image classification framework

Guo, Yanming; Liu, Yu; Bakker, Erwin M.; Guo, Yuanhao; Lew, Michael S.

doi:10.1007/s11042-017-5443-x

Cited by 112 publications

(60 citation statements)

References 37 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, this results in a very complex training and inference procedure. Guo et al (2018) propose a CNN-RNN (recurrent neural network) strategy to address hierarchical classification. A CNN acts as a feature extractor and is trained to predict class labels at the coarse semantic level.…”

Section: Related Workmentioning

confidence: 99%

Exploring Semantic Relationships for Hierarchical Land Use Classification Based on Convolutional Neural Networks

Chun

Rottensteiner

Heipke

2020

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

Abstract. Land use (LU) is an important information source commonly stored in geospatial databases. Most current work on automatic LU classification for updating topographic databases considers only one category level (e.g. residential or agricultural) consisting of a small number of classes. However, LU databases frequently contain very detailed information, using a hierarchical object catalogue where the number of categories differs depending on the hierarchy level. This paper presents a method for the classification of LU on the basis of aerial images that differentiates a fine-grained class structure, exploiting the hierarchical relationship between categories at different levels of the class catalogue. Starting from a convolutional neural network (CNN) for classifying the categories of all levels, we propose a strategy to simultaneously learn the semantic dependencies between different category levels explicitly. The input to the CNN consists of aerial images and derived data as well as land cover information derived from semantic segmentation. Its output is the class scores at three different semantic levels, based on which predictions that are consistent with the class hierarchy are made. We evaluate our method using two test sites and show how the classification accuracy depends on the semantic category level. While at the coarsest level, an overall accuracy in the order of 90% can be achieved, at the finest level, this accuracy is reduced to around 65%. Our experiments also show which classes are particularly hard to differentiate.

show abstract

Section: Related Workmentioning

confidence: 99%

Exploring Semantic Relationships for Hierarchical Land Use Classification Based on Convolutional Neural Networks

Chun

Rottensteiner

Heipke

2020

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…In the early stages of this field, some people tried to combine CNNs and RNNs to solve the general image classification problem. The most straightforward method is to use a CNN to extract histology image features and then input the extracted features into an RNN [5][6][7]. The method proposed by Zheng et al [8] and the method proposed by Nahid et al [9] flatten the feature of the last layer of the convolutional neural network to one dimension and then input the result into the recurrent neural network.…”

Section: Introductionmentioning

confidence: 99%

Parallel Structure Deep Neural Network Using CNN and RNN with an Attention Mechanism for Breast Cancer Histology Image Classification

Yao

Zhang

Zhou

et al. 2019

Cancers

View full text Add to dashboard Cite

In this paper, we present a new deep learning model to classify hematoxylin–eosin-stained breast biopsy images into four classes (normal tissues, benign lesions, in situ carcinomas, and invasive carcinomas). Our model uses a parallel structure consist of a convolutional neural network (CNN) and a recurrent neural network (RNN) for image feature extraction, which is greatly different from the common existed serial method of extracting image features by CNN and then inputting them into RNN. Then, we introduce a special perceptron attention mechanism, which is derived from the natural language processing (NLP) field, to unify the features extracted by the two different neural network structures of the model. In the convolution layer, general batch normalization is replaced by the new switchable normalization method. And the latest regularization technology, targeted dropout, is used to substitute for the general dropout in the last three fully connected layers of the model. In the testing phase, we use the model fusion method and test time augmentation technology on three different datasets of hematoxylin–eosin-stained breast biopsy images. The results demonstrate that our model significantly outperforms state-of-the-art methods.

show abstract

“…An and Liu [13] proposed a CNN–LSTM–support vector machine (SVM) hybrid method for facial expression recognition. In [14], Guo et al performed image classification with a CNN–RNN hybrid model. In [15], Lakhal et al employed CNN architecture to encode remote sensing images into a feature space, and then used LSTM architecture to decode these features to predict the class labels.…”

Section: Introductionmentioning

confidence: 99%

A Novel Method for Classifying Liver and Brain Tumors Using Convolutional Neural Networks, Discrete Wavelet Transform and Long Short-Term Memory Networks

Kutlu

Avcı

2019

Sensors

View full text Add to dashboard Cite

Rapid classification of tumors that are detected in the medical images is of great importance in the early diagnosis of the disease. In this paper, a new liver and brain tumor classification method is proposed by using the power of convolutional neural network (CNN) in feature extraction, the power of discrete wavelet transform (DWT) in signal processing, and the power of long short-term memory (LSTM) in signal classification. A CNN–DWT–LSTM method is proposed to classify the computed tomography (CT) images of livers with tumors and to classify the magnetic resonance (MR) images of brains with tumors. The proposed method classifies liver tumors images as benign or malignant and then classifies brain tumor images as meningioma, glioma, and pituitary. In the hybrid CNN–DWT–LSTM method, the feature vector of the images is obtained from pre-trained AlexNet CNN architecture. The feature vector is reduced but strengthened by applying the single-level one-dimensional discrete wavelet transform (1-D DWT), and it is classified by training with an LSTM network. Under the scope of the study, images of 56 benign and 56 malignant liver tumors that were obtained from Fırat University Research Hospital were used and a publicly available brain tumor dataset were used. The experimental results show that the proposed method had higher performance than classifiers, such as K-nearest neighbors (KNN) and support vector machine (SVM). By using the CNN–DWT–LSTM hybrid method, an accuracy rate of 99.1% was achieved in the liver tumor classification and accuracy rate of 98.6% was achieved in the brain tumor classification. We used two different datasets to demonstrate the performance of the proposed method. Performance measurements show that the proposed method has a satisfactory accuracy rate at the liver tumor and brain tumor classifying.

show abstract

CNN-RNN: a large-scale hierarchical image classification framework

Cited by 112 publications

References 37 publications

Exploring Semantic Relationships for Hierarchical Land Use Classification Based on Convolutional Neural Networks

Exploring Semantic Relationships for Hierarchical Land Use Classification Based on Convolutional Neural Networks

Parallel Structure Deep Neural Network Using CNN and RNN with an Attention Mechanism for Breast Cancer Histology Image Classification

A Novel Method for Classifying Liver and Brain Tumors Using Convolutional Neural Networks, Discrete Wavelet Transform and Long Short-Term Memory Networks

Contact Info

Product

Resources

About