Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning

Gated Convolutional Networks for Cloud Removal From Bi-Temporal Remote Sensing Images

“…Different from shallow learning technology, such as sparse representation and ELM, deep learning, as a powerful representation learning method with deep neural layers, has been widely introduced to image restoration for denoising, deblurring, super-resolution reconstruction, and cloud removal, the latter of which is, in essence, a missing information reconstruction problem. Zhang et al [36] introduced convolutional neural networks (CNNs) to the different tasks of missing information reconstruction and proposed a unified spatial-temporal-spectral framework, which recently was expanded into a spatial-temporal patch-based cloud removal method [37]. Praveer et al [38] applied generative adversarial networks (GANs) to learn the mapping between cloudy images and cloudless images.…”

“…Although the recent deep-learning-based methods have boosted the study of cloud removal and represent the state-of-the-art, some critical points have not yet been addressed, specifically, several useful human insights raised from previous conventional studies are not yet reflected in a current deep-learning framework. The designed cloud removal networks resemble the basic and commonly-used convolutional networks, such as a series of plain convolutional layers [36,37,39] or U-Net [40,41], all of which lack deeper consideration of the specific cloud removal task (i.e., a local-region reconstruction problem). On the one hand, all these deep-learning-based methods [36][37][38][39][40][41] did not discriminate between cloud and cloudless regions and used the same convolution operations to extract layers of features without considering the difference between clouds and clean pixels.…”

“…Although conventional studies tended to use different empirical features to represent clouds and other regions, it is not exploited in the recent deep-learning-based methods. On the other hand, all these methods [36][37][38][39][40][41] used the single pixel-based loss function at the output space. This type of loss function is commonly used in image semantic segmentation; however, it is insufficient for cloud removal from temporal images for the following reasons.…”

“…The network learns the spatiotemporal features from bi-temporal images, a cloudy image, and a clean historical image through a series of gated convolutional layers, which are introduced into cloud removal for the first time in our method. The gated convolutional layers can discriminatively filter out the invalid pixels and encode the abstracted features only from clean pixels for subsequent image repairing, which solves the problem that existed in all the recent deep learning based cloud removal methods [36][37][38][39][40][41] that a common convolutional layer unavoidably encodes information from invalid pixels that would harm the image repairing process. Another notable point is that we use a self-training strategy similar to the work of reference [41] that requires no real training samples but rather requires only clean pixel pairs of bi-temporal images for model training, which is different from [36][37][38][39][40] where real samples had to be prepared through costly and tedious manual work.…”

DRCS-SR: Deep Robust Compressed Sensing for Single Image Super-Resolution

“…However, numerous image super-resolution (SR) methods have been reported to deal with this non-trivial problem [3], [4]. Over the last decade, a various traditional non deeplearning (DL) based approaches have been utilized in SISR computer vision task, including prediction-based methods [5]- [7], edge-based methods [8], [9], statistical methods [10], [11], patch-based methods [12], [13], missing data reconstruction in remote sensing image [14]- [16],and sparse representation methods [10], [17].…”