Enhanced Multi-Stream Remote Sensing Spatiotemporal Fusion Network Based on Transformer and Dilated Convolution

Abstract

Remote sensing images with high temporal and spatial resolutions play a crucial role in land surface-change monitoring, vegetation monitoring, and natural disaster mapping. However, existing technical conditions and cost constraints make it very difficult to directly obtain remote sensing images with high temporal and spatial resolution. Consequently, spatiotemporal fusion technology for remote sensing images has attracted considerable attention. In recent years, deep learning-based fusion methods have been developed. In this study, to improve the accuracy and robustness of deep learning models and better extract the spatiotemporal information of remote sensing images, the existing multi-stream remote sensing spatiotemporal fusion network MSNet is improved using dilated convolution and an improved transformer encoder to develop an enhanced version called EMSNet. Dilated convolution is used to extract time information and reduce parameters. The improved transformer encoder is improved to further adapt to image-fusion technology and effectively extract spatiotemporal information. A new weight strategy is used for fusion that substantially improves the prediction accuracy of the model, image quality, and fusion effect. The superiority of the proposed approach is confirmed by comparing it with six representative spatiotemporal fusion algorithms on three disparate datasets. Compared with MSNet, EMSNet improved SSIM by 15.3% on the CIA dataset, ERGAS by 92.1% on the LGC dataset, and RMSE by 92.9% on the AHB dataset.

LPNet: A remote sensing scene classification method based on large kernel convolution and parameter fusion

“…Li et al. [32] achieved superior feature characterization through the use of dilation convolution which extracts spatiotemporal information of remote sensing images while also reducing model parameters. Wang et al.…”

“…Researchers have utilized various improved convolutional methods to better characterize remote sensing scene images, improving classification accuracy. Li et al [32] achieved superior feature characterization through the use of dilation convolution which extracts spatiotemporal information of remote sensing images while also reducing model parameters. Wang et al [1] utilized depth-separable convolution to extract depth features from remote sensing scene images, achieving highly accurate classification.…”

Advanced Machine Learning and Deep Learning Approaches for Remote Sensing

“…However, it is difficult to directly obtain high-resolution remote sensing images, and thus the deployment of space-time convergence technology to obtain remote sensing images is receiving a lot of attention. In the contribution by Li et al, entitled "Enhanced Multi-Stream Remote Sensing Spatiotemporal Fusion Network Based on Transformer and Dilated Convolution", the authors propose a deep learning model with high accuracy and robustness to better extract spatiotemporal information from remote sensing images [16]. The proposed model is EMSNet, which extends the existing MSNet.…”