Applying self-supervised learning for semantic cloud segmentation of all-sky images

Abstract

Abstract. Semantic segmentation of ground-based all-sky images (ASIs) can provide high-resolution cloud coverage information of distinct cloud types, applicable for meteorology-, climatology- and solar-energy-related applications. Since the shape and appearance of clouds is variable, and there is high similarity between cloud types, a clear classification is difficult. Therefore, most state-of-the-art methods focus on the distinction between cloudy and cloud-free pixels without taking into account the cloud type. On the other hand, cloud classification is typically determined separately at the image level, neglecting the cloud's position and only considering the prevailing cloud type. Deep neural networks have proven to be very effective and robust for segmentation tasks; however they require large training datasets to learn complex visual features. In this work, we present a self-supervised learning approach to exploit many more data than in purely supervised training and thus increase the model's performance. In the first step, we use about 300 000 ASIs in two different pretext tasks for pretraining. One of them pursues an image reconstruction approach. The other one is based on the DeepCluster model, an iterative procedure of clustering and classifying the neural network output. In the second step, our model is fine-tuned on a small labeled dataset of 770 ASIs, of which 616 are used for training and 154 for validation. For each of them, a ground truth mask was created that classifies each pixel into clear sky or a low-layer, mid-layer or high-layer cloud. To analyze the effectiveness of self-supervised pretraining, we compare our approach to randomly initialized and pretrained ImageNet weights using the same training and validation sets. Achieving 85.8 % pixel accuracy on average, our best self-supervised model outperforms the conventional approaches of random (78.3 %) and pretrained ImageNet initialization (82.1 %). The benefits become even more evident when regarding precision, recall and intersection over union (IoU) of the respective cloud classes, where the improvement is between 5 and 20 percentage points. Furthermore, we compare the performance of our best model with regards to binary segmentation with a clear-sky library (CSL) from the literature. Our model outperforms the CSL by over 7 percentage points, reaching a pixel accuracy of 95 %.

Validation of a camera-based intra-hour irradiance nowcasting model using synthetic cloud data

“…We chose the DeeplabV3+ (Chen et al, 2018) CNN architecture which is designed using an Encoder-Decoder structure as also the UNet (Ronneberger et al, 2015) used by Fabel et al (2022) does. For the encoder, we use a ResNet34 (He et al, 2015) pre-trained on the Imagenet dataset (Russakovsky et al, 2014).…”

“…The most important information to obtain from all-sky images is the classification of pixels as cloudy or clear. Convolutional neural networks (CNN), which are commonly applied for image segmentation have also been applied to images of clouds to generate cloudmasks (e.g., Dev et al, 2019;Xie et al, 2020;Fabel et al, 2022). Also our cloudmask derivation relies on CNN, we used the DeeplabV3+ network structure (Chen et al, 2018).…”

“…Also convolutional neural networks (CNN) have been proven to work beneficially for these tasks (Ye et al, 2017;Dev et al, 2019;Xie et al, 2020;Hasenbalg et al, 2020) when trained on densely labeled data. Fabel et al (2022) demonstrated the use of a CNN to distinguish not only clear and cloudy pixels, but further separated clouds into three subclasses for low, mid and high layer clouds. Further on, projected cloudmasks of multiple imagers onto a common plane and combined them for an analysis of spatial variations of irradiances.…”

Obscurant Segmentation in Long Wave Infrared Images Using GLCM Textures

“…For verification, the efficacy of specific color channels in detecting cloud pixels, 1D and 2D clustering results are presented across all 16 color channels for all the images of HYTA and evaluated using precision, recall, and F-score. In [ 21 , 22 , 23 ], the authors propose a deep learning based approaches to segment sky/cloud images. The model proposed in 2019 in [ 21 ] achieved a binary classification accuracy of , the proposed approach from 2019 in [ 22 ] achieved a accuracy, and [ 23 ] published in 2022 achieved precision.…”

“…In [ 21 , 22 , 23 ], the authors propose a deep learning based approaches to segment sky/cloud images. The model proposed in 2019 in [ 21 ] achieved a binary classification accuracy of , the proposed approach from 2019 in [ 22 ] achieved a accuracy, and [ 23 ] published in 2022 achieved precision. However, similar to the previously listed approaches, the network architecture proposed is designed using visible light images (RGB channels).…”