A CNN–RNN architecture for multi-label weather recognition

Abstract

Weather Recognition plays an important role in our daily lives and many computer vision applications. However, recognizing the weather conditions from a single image remains challenging and has not been studied thoroughly. Generally, most previous works treat weather recognition as a single-label classification task, namely, determining whether an image belongs to a specific weather class or not. This treatment is not always appropriate, since more than one weather conditions may appear simultaneously in a single image. To address this problem, we make the first attempt to view weather recognition as a multi-label classification task, i.e., assigning an image more than one labels according to the displayed weather conditions. Specifically, a CNN-RNN based multi-label classification approach is proposed in this paper. The convolutional neural network (CNN) is extended with a channel-wise attention model to extract the most correlated visual features. The Recurrent Neural Network (RNN) further processes the features and excavates the dependencies among weather classes.Finally, the weather labels are predicted step by step. Besides, we construct two datasets for the weather recognition task and explore the relationships among different weather conditions. Experimental results demonstrate the superiority and effectiveness of the proposed approach. The new constructed datasets will be available at https: //github.com/wzgwzg/Multi-Label-Weather-Recognition.

WeatherNet: Recognising Weather and Visual Conditions from Street-Level Images Using Deep Residual Learning

“…To provide a quantitative comparison, we applied WeatherNet to two open-sourced datasets used in previous studies [29,41]. Table 5 describes the datasets used for evaluation, in terms of size, labels, and the original approach used for prediction.…”

“…While this model shows progress in recognising more weather classes, it only sees weather conditions as exclusive classes, ignoring the co-existence of two or more classes in a single image for a given time. Finally, to solve the combination issue of the existence of multiple weather class in a single image, [29] used a CNN based model that includes an attention-layer to allow the model to infer more than a class for a given time depending on the characteristics of the input image. While this model shows progress in classifying multiple weather conditions and their combinations (sunny, cloudy, foggy, rainy, and snowy), it still ignores the dynamics of visual conditions and the time of day that may influence weather classification accuracy.…”

“…This discussion is partly qualitative due to the absence of benchmark datasets to compare the results from all the methods. However, we also evaluated the performance of WeatherNet on two available datasets, [29,41], and compared the results of our framework with the original outputs.…”

Precipitation prediction using recurrent neural networks and long short-term memory

“…Some methods that are often used in deep learning are convolutional neural network (CNN), which convolutes with a fixed size kernel. Weather data can be viewed as imagery, so it can be resolved with CNN [23]. However, for a limited image segment, it indeed will collide with memory limitations, so other methods are needed.…”

Parallel Structure Deep Neural Network Using CNN and RNN with an Attention Mechanism for Breast Cancer Histology Image Classification

“…Hua's [10] method is similar to Zheng's. Zhao et al introduced channel-wise attention [11]. Koo et al [12] used the features extracted from the intermediate convolution layer as input to the recurrent neural network.…”