A Novel Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions Using a Two-Stream Convolutional Neural Network

Abstract

Spatiotemporal fusion is considered a feasible and cost-effective way to solve the trade-off between the spatial and temporal resolution of satellite sensors. Recently proposed learning-based spatiotemporal fusion methods can address the prediction of both phenological and land-cover change. In this paper, we propose a novel deep learning-based spatiotemporal data fusion method that uses a two-stream convolutional neural network. The method combines both forward and backward prediction to generate a target fine image, where temporal change-based and a spatial information-based mapping are simultaneously formed, addressing the prediction of both phenological and land-cover changes with better generalization ability and robustness. Comparative experimental results for the test datasets with phenological and land-cover changes verified the effectiveness of our method. Compared to existing learning-based spatiotemporal fusion methods, our method is more effective in predicting phenological change and directly reconstructing the prediction with complete spatial details without the need for auxiliary modulation.

A Hybrid Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions

“…Differently from existing DL-based STF methods that need at least two pairs of finecoarse images [42,48,[53][54][55], HDLSFM requires minimal input data, i.e., a known fine ( ) and a coarse ( ) image at a base date ( ), and a coarse image ( ) at a prediction date ( ), to generate a robust target fine image ( ) containing both PC and LC information. This is achieved through a hybrid framework consisting of a nonlinear DL-based relative radiometric normalization that alleviates radiation differences, a DL-based SR algorithm for LC prediction, and linear-based fusion for PC prediction.…”

“…Note that an aggregation step before the nonlinear mapping formulation is necessary, since a direct establishment of the mapping between the original fine and coarse images may not be effective due to the significant magnification, and may introduce uncertainties [42,53]. Instead, by aggregating the spatial resolution of the fine image to reduce the magnification, the reliability of the nonlinear mapping is improved.…”

“…In order to convert the spatial resolution of image ( ) from low to high, spatial interpolation, such as thin plate spline interpolation, has been widely utilized in STF methods [12,44]. In this paper, the DL-based SR method is used to retain the complementary LC information, which is considered to be more beneficial for accurate LC prediction [36], and underpins a higher generalization ability because of the introduction of prior information [53,55]. According to the core of the DL-based SR method, the nonlinear mapping between and is formulated as:…”

“…This study site was chosen to test the effectiveness of HDLSFM in PC prediction. The dataset of this experiment was the same as the one used by Jia et al [53].…”

“…However, no specific temporal change information is utilized, resulting in limited PC prediction capabilities. In order to improve the DL-based STF methods' PC prediction capabilities, temporal change information in DL-based STF methods was introduced in [53], through the formulation of a temporal change-based mapping. However, this method requires two pairs of known fine and coarse images to establish the mapping parameters, which limits its applicability in areas with severe cloud contamination.…”