A Hybrid Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions

Abstract

Spatiotemporal fusion (STF) is considered a feasible and cost-effective way to deal with the trade-off between the spatial and temporal resolution of satellite sensors, and to generate satellite images with high spatial and high temporal resolutions. This is achieved by fusing two types of satellite images, i.e., images with fine temporal but rough spatial resolution, and images with fine spatial but rough temporal resolution. Numerous STF methods have been proposed, however, it is still a challenge to predict both abrupt landcover change, and phenological change, accurately. Meanwhile, robustness to radiation differences between multi-source satellite images is crucial for the effective application of STF methods. Aiming to solve the abovementioned problems, in this paper we propose a hybrid deep learning-based STF method (HDLSFM). The method formulates a hybrid framework for robust fusion with phenological and landcover change information with minimal input requirements, and in which a nonlinear deep learning-based relative radiometric normalization, a deep learning-based superresolution, and a linear-based fusion are combined to address radiation differences between different types of satellite images, landcover, and phenological change prediction. Four comparative experiments using three popular STF methods, i.e., spatial and temporal adaptive reflectance fusion model (STARFM), flexible spatiotemporal data fusion (FSDAF), and Fit-FC, as benchmarks demonstrated the effectiveness of the HDLSFM in predicting phenological and landcover change. Meanwhile, HDLSFM is robust for radiation differences between different types of satellite images and the time interval between the prediction and base dates, which ensures its effectiveness in the generation of fused time-series data.

A Pseudo-Siamese Deep Convolutional Neural Network for Spatiotemporal Satellite Image Fusion

“…The original image size is 1720 × 2040, and the main characteristic of the CIA is that the spatial information is more complex. The temporal change information is primarily concentrated in the seasonal phenology change, and the change in land-cover type is less [25].…”

“…Typical methods include the flexible spatiotemporal method (FSDAF) [21], improved FSDAF [22], enhanced FSDAF [23], and FSDAF 2.0 [24]. This method can effectively balance the spatial detail preservation and spectral change reconstruction, but the high complexity of the algorithm limits its wider application [25].…”

“…Learning-based methods have developed rapidly in recent years and have been widely used in the field of remote sensing image processing [25], [26]. The main idea of this method is to use numerous known data samples to train the weight parameters of the model [27], [28], automatically learning the feature information from the known data samples and inputting the coarse image of the prediction time in the trained model to generate a fine image at the prediction time.…”

“…The convolutional neural network (CNN) is generally used to establish nonlinear mapping between the input and output data using the deep learning method. Numerous research methods have proved that the CNN can be effectively applied in the field of remote sensing data fusion [11], [25], [30].…”

On-the-Fly Fusion of Remotely-Sensed Big Data Using an Elastic Computing Paradigm with a Containerized Spark Engine on Kubernetes

“…GPU-accelerated image processing libraries such as CLIJ are widely used [ 17 ]. Deep learning-based spatiotemporal fusion methods can thus be significantly accelerated via the powerful computational abilities of GPUs [ 18 ]. Recently, Hong proposed a novel extension of Spark, which offers GPU-accelerated scalable computing [ 19 ].…”