STF-EGFA: A Remote Sensing Spatiotemporal Fusion Network with Edge-Guided Feature Attention

Abstract

Spatiotemporal fusion in remote sensing plays an important role in Earth science applications by using information complementarity between different remote sensing data to improve image performance. However, several problems still exist, such as edge contour blurring and uneven pixels between the predicted image and the real ground image, in the extraction of salient features by convolutional neural networks (CNNs). We propose a spatiotemporal fusion method with edge-guided feature attention based on remote sensing, called STF-EGFA. First, an edge extraction module is used to maintain edge details, which effectively solves the boundary blurring problem. Second, a feature fusion attention module is used to make adaptive adjustments to the extracted features. Among them, the spatial attention mechanism is used to solve the problem of weight variation in different channels of the network. Additionally, the problem of uneven pixel distribution is addressed with a pixel attention (PA) mechanism to highlight the salient features. We transmit the different features extracted by the edge module and the encoder to the feature attention (FA) module at the same time after the union. Furthermore, the weights of edges, pixels, channels and other features are adaptively learned. Finally, three remote sensing spatiotemporal fusion datasets, Ar Horqin Banner (AHB), Daxing and Tianjin, are used to verify the method. Experiments proved that the proposed method outperformed three typical comparison methods in terms of the overall visual effect and five objective evaluation indexes: spectral angle mapper (SAM), peak signal-to-noise ratio (PSNR), spatial correlation coefficient (SCC), structural similarity (SSIM) and root mean square error (RMSE). Thus, the proposed spatiotemporal fusion algorithm is feasible for remote sensing analysis.

Color Night Light Remote Sensing Images Generation Using Dual-Transformation

“…As both the component and the panchromatic image can reflect the gray change in ground objects, the component is expressed in the panchromatic image to obtain the component with more detailed features. Finally, the new RGB image is obtained using the IHS inverse transformation, as shown in Equation (4) follows [ 25 ]: where , and denote the red, green, and blue new bands of the multi-spectral image; and denote the intermediate variables of RGB conversion to the IHS color space; denotes the new value of after the IHS transformation.…”

“…As both the component I and the panchromatic image can reflect the gray change in ground objects, the component I is expressed in the panchromatic image to obtain the component I new with more detailed features. Finally, the new RGB image is obtained using the IHS inverse transformation, as shown in Equation ( 4) follows [25]:…”

Applicability Analysis of GF-2PMS and PLANETSCOPE Data for Ground Object Recognition in Karst Region

“…The visual interpretation method can directly analyze the similarity between the fused image and the real image and yield a preliminary judgment on the fusion accuracy of each model. The correlation analysis method mainly uses four evaluation metrics: average absolute deviation (AAD), root mean square error (RMSE), correlation coefficient (CC), and structural similarity (SSIM) [26,46,47]. These indexes are used to quantitatively evaluate the similarity between the fused image and the real image.…”

An Experimental Study of the Accuracy and Change Detection Potential of Blending Time Series Remote Sensing Images with Spatiotemporal Fusion

“…These models contributed to building a common framework for spationtemporal fusion algorithms that employs the use of two streams and the stepwise modeling of spatial, sensor, and temporal differences. In recent works [8][9][10][11][12][13][14][15], multiscale learning, spatial channel attention mechanisms, and edge reservation have been introduced into CNNs for the extraction and integration of features.…”