Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks

Liu, Shuang; Duan, Linlin; Zhang, Zhong; Cao, Xiaozhong

doi:10.1109/access.2019.2926092

Cited by 22 publications

(25 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liu. et al [17] proposed a method integrates the high-level fusion and the output of low-level fusion with deep visual features and deep multimodal features. Shi.…”

Section: Related Workmentioning

confidence: 99%

Notice of Violation of IEEE Publication Principles: Ground-Based Cloud Image Recognition System Based on Multi-CNN and Feature Screening and Fusion

Zhang

Jing

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The recognition of ground-based cloud images has rich application prospects in many aspects such as weather prediction, astronomical site selection, and meteorological observation. Affected by factors such as rotation and illumination, the traditional feature extraction method is difficult to accurately describe the features of cloud images, resulting in low accuracy of ground-based cloud image recognition, and cannot meet the requirements of practical applications. With the popularity of convolutional neural networks in image processing, ground-based cloud image recognition algorithms based on convolutional neural network have become a research focus. However, the features of the ground-based cloud image are relatively shallow, and the cloud texture and other features are seriously lost in the convolution process, and it is difficult to achieve a good recognition effect. This paper proposes a ground-based cloud image recognition system based on multi-scale convolutional neural network (Multi-CNN) and multilayer perceptron neural networks (MLP). The multi-level and multi-scale convolution feature extraction is performed through convolution layers of Multi-CNN, and the local features with strong resolving power are selected through the feature screening algorithm based on DP clustering. Finally, the local features are encoded and fused for cloud image classification based on MLP. Filed test results show that our method was superior to other tested network models in terms of the recognition accuracy of 94.8% under 9 classification. In addition, ablation experiments show that the multi-scale feature extraction, screening and local feature coding in this paper have a significant effect on improving the algorithm's ability to distinguish different cloud images. INDEX TERMS Ground-based cloud image; CNN; cloud classification.

show abstract

“…Liu. et al [17] proposed a method integrates the high-level fusion and the output of low-level fusion with deep visual features and deep multimodal features. Shi.…”

Section: Related Workmentioning

confidence: 99%

Notice of Violation of IEEE Publication Principles: Ground-Based Cloud Image Recognition System Based on Multi-CNN and Feature Screening and Fusion

Zhang

Jing

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…They obtained the classification result of 86.48% over 8000 ground-based cloud samples. Liu et al [63] fused deep multimodal and deep visual features in a two-level fashion, i.e., low-level and high-level. The low-level fused the heterogeneous features directly and its output was regarded as a part of the input of the high-level which also integrates deep visual and deep multimodal features.…”

Section: Overall Discussionmentioning

confidence: 99%

“…The comparison results between the proposed MMFN and other methods, such as [32,62,63], are summarized in Table 2. Firstly, most results in the right part of the table are more competitive than those in the left part, which indicates that the multi-modal information contains useful information for ground-based cloud recognition.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…Such a strategy thoroughly investigates the correlations between the visual features and the multi-modal information and takes into consideration the complementary and supplementary information between them as well as their relative importance for the recognition task. 77.95 DMF [31] 79.05 DCAFs [25] 82.67 DCAFs + MI 82.97 CloutNet [26] 79.92 CloutNet + MI 80.37 JFCNN [32] 84.13 DTFN [62] 86.48 HMF [63] 87.90 MMFN 88.63…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

et al. 2020

Self Cite

View full text Add to dashboard Cite

In recent times, deep neural networks have drawn much attention in ground-based cloud recognition. Yet such kind of approaches simply center upon learning global features from visual information, which causes incomplete representations for ground-based clouds. In this paper, we propose a novel method named multi-evidence and multi-modal fusion network (MMFN) for ground-based cloud recognition, which could learn extended cloud information by fusing heterogeneous features in a unified framework. Namely, MMFN exploits multiple pieces of evidence, i.e., global and local visual features, from ground-based cloud images using the main network and the attentive network. In the attentive network, local visual features are extracted from attentive maps which are obtained by refining salient patterns from convolutional activation maps. Meanwhile, the multi-modal network in MMFN learns multi-modal features for ground-based cloud. To fully fuse the multi-modal and multi-evidence visual features, we design two fusion layers in MMFN to incorporate multi-modal features with global and local visual features, respectively. Furthermore, we release the first multi-modal ground-based cloud dataset named MGCD which not only contains the ground-based cloud images but also contains the multi-modal information corresponding to each cloud image. The MMFN is evaluated on MGCD and achieves a classification accuracy of 88.63% comparative to the state-of-the-art methods, which validates its effectiveness for ground-based cloud recognition.

show abstract

“…Recently, plenty of works (Shi et al, 2017;Ye et al, 2017) have obtained encouraging results by extracting the cloud signature from pre-trained CNNs, such as AlexNet (Krizhevsky et al, 2012) and VGGNet (Simonyan and Zisserman, 2015). In addition, attempts have been made to simply exploit end-to-end CNN models for cloud categorization (Li et al, 2020;Liu et al, 2019;Zhang et al, 2018b). However, the insufficiency of labelled samples might make the network hard to converge in the training stage.…”

mentioning

confidence: 99%

Improving Cloud Type Classification of Ground-Based Images Using Region Covariance Descriptors

Tang¹,

Yang²,

Zhou³

et al. 2020

Preprint

View full text Add to dashboard Cite

Abstract. Cloud types are important indicators of cloud characteristics and short-term weather forecasting. The meteorological researchers can benefit from the automatic cloud type recognition of massive images captured by the ground-based imagers. However, by far it is still of huge challenge to design a powerful discriminative classifier for cloud categorization. To tackle this difficulty, in this paper, we present an improved method with region covariance descriptors (RCovDs) and Riemannian Bag-of-Feature (BoF). RCovDs model the correlations among different dimensional features, that allows for a more discriminative representation. BoF is extended from Euclidean space to Riemannian manifold by k-Means clustering, in which Stein divergence is adopted as a similarity metric. The histogram feature is extracted by encoding RCovDs of the cloud image blocks with BoF-based codebook. The multi-class support vector machine (SVM) is utilized for the recognition of cloud types. The experiments on the ground-based cloud image datasets validate the proposed method and exhibit the competitive performance against state-of-the-art methods.

show abstract

Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks

Cited by 22 publications

References 27 publications

Notice of Violation of IEEE Publication Principles: Ground-Based Cloud Image Recognition System Based on Multi-CNN and Feature Screening and Fusion

Notice of Violation of IEEE Publication Principles: Ground-Based Cloud Image Recognition System Based on Multi-CNN and Feature Screening and Fusion

Multi-Evidence and Multi-Modal Fusion Network for Ground-Based Cloud Recognition

Improving Cloud Type Classification of Ground-Based Images Using Region Covariance Descriptors

Contact Info

Product

Resources

About