<i>StfNet</i>: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion

Liu, Xun; Deng, Chenwei; Chanussot, Jocelyn; Hong, Danfeng; Zhao, Baojun

doi:10.1109/tgrs.2019.2907310

Cited by 144 publications

(82 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A possible solution is to further improve learning-based models. Recent studies [26][27][28] have applied CNNs for spatiotemporal fusion, and obtained better results in shape change monitoring. Many spatiotemporal fusion models based on deep learning use super-resolution reconstruction strategies.…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

Liu

Yin

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

In recent years, many spatial and temporal satellite image fusion (STIF) methods have been developed to solve the problems of trade-off between spatial and temporal resolution of satellite sensors. This study, for the first time, conducted both scene-level and local-level comparison of five state-of-art STIF methods from four categories over landscapes with various spatial heterogeneity and temporal variation. The five STIF methods include the spatial and temporal adaptive reflectance fusion model (STARFM) and Fit-FC model from the weight function-based category, an unmixing-based data fusion (UBDF) method from the unmixing-based category, the one-pair learning method from the learning-based category, and the Flexible Spatiotemporal DAta Fusion (FSDAF) method from hybrid category. The relationship between the performances of the STIF methods and scene-level and local-level landscape heterogeneity index (LHI) and temporal variation index (TVI) were analyzed. Our results showed that (1) the FSDAF model was most robust regardless of variations in LHI and TVI at both scene level and local level, while it was less computationally efficient than the other models except for one-pair learning; (2) Fit-FC had the highest computing efficiency. It was accurate in predicting reflectance but less accurate than FSDAF and one-pair learning in capturing image structures; (3) One-pair learning had advantages in prediction of large-area land cover change with the capability of preserving image structures. However, it was the least computational efficient model; (4) STARFM was good at predicting phenological change, while it was not suitable for applications of land cover type change; (5) UBDF is not recommended for cases with strong temporal changes or abrupt changes. These findings could provide guidelines for users to select appropriate STIF method for their own applications.

show abstract

Section: Limitations and Future Directionsmentioning

confidence: 99%

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

Liu

Yin

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The ground truth is a local climate zone label released by the IEEE GRSS IADF for the data fusion contest in 2017. 4 3) Label configuration For both the LCLU data set and the LCZ data set, as shown in Fig. 3 and Fig.…”

Section: Experiments and Discussionmentioning

confidence: 99%

MIMA: MAPPER-Induced Manifold Alignment for Semi-Supervised Fusion of Optical Image and Polarimetric SAR Data

Hong

Zhu

2019

IEEE Trans. Geosci. Remote Sensing

Self Cite

View full text Add to dashboard Cite

This is the pre-acceptance version, to read the final version please go to IEEE Transactions on Geoscience and Remote Sensing on IEEE Xplore. Multi-modal data fusion has recently been shown promise in classification tasks in remote sensing. Optical data and radar data, two important yet intrinsically different data sources, are attracting more and more attention for potential data fusion. It is already widely known that, a machine learning based methodology often yields excellent performance. However, the methodology relies on a large training set, which is very expensive to achieve in remote sensing. The semi-supervised manifold alignment (SSMA), a multi-modal data fusion algorithm, has been designed to amplify the impact of an existing training set by linking labeled data to unlabeled data via unsupervised techniques. In this paper, we explore the potential of SSMA in fusing optical data and polarimetric SAR data, which are multi-sensory data sources. Furthermore, we propose a MAPPER-induced manifold alignment (MIMA) for semisupervised fusion of multi-sensory data sources. Our proposed method unites SSMA with MAPPER, which is developed from the emerging topological data analysis (TDA) field. To our best knowledge, this is the first time that SSMA has been applied on fusing optical data and SAR data, and also the first time that TDA has been applied in remote sensing. The conventional SSMA derives a topological structure using k-nearest-neighbor (kNN), while MIMA employs MAPPER, which considers the field knowledge and derives a novel topological structure through the spectral clustering in a data-driven fashion. Experiment results on data fusion with respect to land cover land use classification and local climate zone classification suggest superior performance of MIMA.Index Terms-Hyperspectral image, MAPPER, multi-modal data fusion, multi-sensory data fusion, multispectral image, Pol-SAR, semi-supervised manifold alignment (SSMA), topological data analysis (TDA).

show abstract

“…We have to emphatically clear that the motivation and goal of this paper apply existing labels from the task-independent dataset to achieve auto-tagging and detection of objects on target dataset, rather than greatly enhancing feature representation capabilities. Therefore, the F-RCN has been only selected as an example, the proposed WSDN could be easily adopted with other networks [18,19].…”

Section: Detector Generationmentioning

confidence: 99%

A Weakly-Supervised Deep Network for DSM-Aided Vehicle Detection

Hong

Tian

et al. 2019

IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium

Self Cite

View full text Add to dashboard Cite

With the breakthrough of the spatial resolution of optical remote sensing images at the sub-meter level and the explosive development of deep learning, geospatial object detection has achieved a growing interest in remote sensing community. However, labeling large training datasets in object level is still an expensive and tedious procedure. This might lead to the poor model generalization and degraded network learning ability. To this end, a weakly-supervised deep network (WSDN) is developed for geospatial object detection by applying a digital surface model (DSM)-aided auto-labeling and a pre-trained network learned from the task-independent dataset. Experimental results conducted on the stereo aerial imagery of a large camping site are performed to demonstrate that the proposed WSDN yields better detection results, with 62.78% precision and 55.13% recall.

show abstract

StfNet: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion

Cited by 144 publications

References 40 publications

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

Comparison of Five Spatio-Temporal Satellite Image Fusion Models over Landscapes with Various Spatial Heterogeneity and Temporal Variation

MIMA: MAPPER-Induced Manifold Alignment for Semi-Supervised Fusion of Optical Image and Polarimetric SAR Data

A Weakly-Supervised Deep Network for DSM-Aided Vehicle Detection

Contact Info

Product

Resources

About