WeatherNet: Recognising Weather and Visual Conditions from Street-Level Images Using Deep Residual Learning

Abstract

Extracting information related to weather and visual conditions at a given time and space is indispensable for scene awareness, which strongly impacts our behaviours, from simply walking in a city to riding a bike, driving a car, or autonomous drive-assistance. Despite the significance of this subject, it has still not been fully addressed by the machine intelligence relying on deep learning and computer vision to detect the multi-labels of weather and visual conditions with a unified method that can be easily used in practice. What has been achieved to-date are rather sectorial models that address a limited number of labels that do not cover the wide spectrum of weather and visual conditions. Nonetheless, weather and visual conditions are often addressed individually. In this paper, we introduce a novel framework to automatically extract this information from street-level images relying on deep learning and computer vision using a unified method without any pre-defined constraints in the processed images. A pipeline of four deep convolutional neural network (CNN) models, so-called WeatherNet, is trained, relying on residual learning using ResNet50 architecture, to extract various weather and visual conditions such as dawn/dusk, day and night for time detection, glare for lighting conditions, and clear, rainy, snowy, and foggy for weather conditions. WeatherNet shows strong performance in extracting this information from user-defined images or video streams that can be used but are not limited to autonomous vehicles and drive-assistance systems, tracking behaviours, safety-related research, or even for better understanding cities through images for policy-makers. environment due to precipitation including clear, rainy, foggy, or snowy weather. They represent crucial factors for many urban studies including transport, behaviour, and safety-related research [5]. For example, walking, cycling, or driving in rainy weather is associated with a higher risk of experiencing an incident than in clear weather [5,6]. Fog, snow, and glare have also been found to increase risk [6,7]. Importantly, it is not only the inherent risk that different weather and visual conditions pose to human life that is of interest to researchers. Scene awareness for autonomous navigation in cities is highly influenced by the dynamics of weather and visual conditions and it is imperative for any vision system to cope with them simultaneously [8]. For example, object detection algorithms must perform well in fog and glare as well as in clear conditions, in order to be reliable. Accordingly, finding an automatic approach to extract this information from images or video streams is in high demand for computer scientists, planners, and policy-makers.While there are different methods that are used to understand the dynamics of weather and visual conditions, a knowledge gap appears when addressing this subject. To date, these two crucial domains-weather and visual conditions-have been studied individually, ignoring the importance of understanding the dy...

Development of a Novel Convolutional Neural Network Architecture Named RoadweatherNet for Trajectory-Level Weather Detection using SHRP2 Naturalistic Driving Data

“…Previous studies have used pre-trained CNN architecture to develop weather detection models. For instance, the study by Ibrahim et al ( 12 ) developed a weather detection model based on ResNet50 architecture. However, pre-trained models, including ResNet50, are computationally expensive because of their complex structure.…”

Weather and Light Level Classification for Autonomous Driving: Dataset, Baseline and Active Learning

“…Previously the two domains were studied individually or as a binary classification task, i.e., belongs to a specific label or not. Deep neural network models outperform all of the previous nominal methods related to Mathematical models, filterbased models, and machine learning models using shallow networks, as discussed by Ibrahim et al [9].…”

Visibility Enhancement and Fog Detection: Solutions Presented in Recent Scientific Papers with Potential for Application to Mobile Systems

“…Remaining on static systems, information about weather and the visual condition is needed in the context of traffic safety topics; therefore, a framework to automatically extract data from street-level images is presented by [ 114 ]. Deep learning and computer vision are used alongside a unified method that has no pre-defined constraints.…”