SEnSeI: A Deep Learning Module for Creating Sensor Independent Cloud Masks

Abstract

We introduce a novel neural network architecture-Spectral ENcoder for SEnsor Independence (SEnSeI)-by which several multispectral instruments, each with different combinations of spectral bands, can be used to train a generalised deep learning model. We focus on the problem of cloud masking, using several pre-existing datasets, and a new, freely available dataset for Sentinel-2. Our model is shown to achieve state-of-the-art performance on the satellites it was trained on (Sentinel-2 and Landsat 8), and is able to extrapolate to sensors it has not seen during training such as Landsat 7, Per úSat-1, and Sentinel-3 SLSTR. Model performance is shown to improve when multiple satellites are used in training, approaching or surpassing the performance of specialised, single-sensor models. This work is motivated by the fact that the remote sensing community has access to data taken with a hugely variety of sensors. This has inevitably led to labelling efforts being undertaken separately for different sensors, which limits the performance of deep learning models, given their need for huge training sets to perform optimally. Sensor independence can enable deep learning models to utilise multiple datasets for training simultaneously, boosting performance and making them much more widely applicable. This may lead to deep learning approaches being used more frequently for on-board applications and in ground segment data processing, which generally require models to be ready at launch or soon afterwards.

An Introduction to Machine and Deep Learning Methods for Cloud Masking Applications

“…It is an example of multi-class segmentation (clear, cloud, cloud shadow, thin cloud, and invalid). KappaMask was trained on KappaZeta Sentinel-2 annotated masks created by the authors and is freely available, and on Sentinel-2 CloudCatalogue [104]. The authors provide a comprehensive analysis of KappaMask's performance compared to state-of-the-art rule-based and Machine Learning methods.…”

“…The previous works have shown the need to adapt the data in terms of spatial and spectral characteristics, e.g., by resampling operations or by discarding some spectral information, which leads to a lack of immediacy in the transfer of models from one sensor to another, as well as a lack of bands that may be meaningful for cloud detection. The sensor dependence of a model is addressed by Francis et al in [119]. The proposed Spectral Encoder for Sensor Independence (SEnSeI) new architecture, represents a notable advancement in remote sensing applications and in cloud masking in particular, functioning as a preprocessing module that translates data from various sensors into a unified format.…”

“…It is common to assess the result of a masking method by measuring its agreement with reference ground truths, which can be manually or tool-supported annotated masks, the most popular of which are summarized in Table 1, or validated cloud masks released as official satellite products. [104] Worldwide 513 KappaZeta [103] Northern Europe 150 CloudSEN12 [122] Worlwide 49,400 WHUS2-CD [123] China 32…”

“…Improvements in transfer learning techniques and data adaptability can benefit to address the problem. An important step forward in this direction is represented by the work in [119], where the sensor independence of a model is highlighted to make better use of existing data, increase the efficiency and applicability of models, and ultimately contribute to more effective and versatile remote sensing analysis.…”

Sensor Independent Cloud and Shadow Masking With Partial Labels and Multimodal Inputs

“…This is because the cycle of model development and validation, and the creation of datasets for training (in the case of supervised models) and validation (for all models) must be repeated for A. Francis is with the Φ-lab, ESRIN, Frascati, Italy each sensor. Having previously introduced sensor independence with Spectral ENcoder for SEnsor Independence (SEn-SeI) [3]-whereby a single model may be trained and used on multiple multispectral sensors-this work extends that effort, creating a cloud masking model which is further generalised than before, whilst achieving state-of-the-art performance.…”