Multi-modal temporal attention models for crop mapping from satellite time series

Garnot, Vivien Sainte Fare; Landrieu, Loïc; Chehata, Nesrine

doi:10.1016/j.isprsjprs.2022.03.012

Cited by 35 publications

(22 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, we report the performance of the existing baselines and attention variations for comparison. Note that due to sampling rate inconsistencies between the modalities, similar to [29], we have interpolated Radar observations to match the time stamps of the multi-spectral modality. Analysing the results presented in Tab.…”

Section: A Results Of the Fusion Modelmentioning

confidence: 99%

“…2) PASTIS-R [29] dataset: PASTIS-R is an extension of the PASTIS dataset where the authors have combined the Sentinel-2 multi-spectral patches with the corresponding Sentinel-1 observations. Specifically, they have used Sentinel-1 data in ground range detected format and assembled these into 3channel images (vertical polarization, horizontal polarization and the ratio of vertical over horizontal polarization).…”

Section: Methodsmentioning

confidence: 99%

“…Separate observations were captured in both ascending and descending orbits, generating two 3-channel images for each observation. This multi-modal dataset has rich spectral information coming from the Sentinel-2 multi-spectral observations, while the Cband radar of Sentinel-1 captures useful geometric information as it's immune to cloud cover [29].…”

Section: Methodsmentioning

confidence: 99%

“…In [29], the authors have introduced several fusion strategies for the PASTIS-R dataset. Their evaluations using early, late, decision, and mid level fusion indicated that early fusion is the best strategy for panoptic segmentation.…”

Section: B Knowledge Distillation Frameworkmentioning

confidence: 99%

“…Then PQ is the product of SQ and RQ (i.e. P R = SQ × RQ), thus combining the detection, classification and delineation accuracies [29].…”

Section: Evaluationsmentioning

confidence: 99%

See 4 more Smart Citations

Towards On-Board Panoptic Segmentation of Multispectral Satellite Images

Fernando¹,

Fookes²,

Gammulle³

et al. 2022

Preprint

View full text Add to dashboard Cite

With tremendous advancements in low-power embedded computing devices and remote sensing instruments, the traditional satellite image processing pipeline which includes an expensive data transfer step prior to processing data on the ground is being replaced by on-board processing of captured data. This paradigm shift enables critical and time-sensitive analytic intelligence to be acquired in a timely manner onboard the satellite itself. However, at present, the on-board processing of multi-spectral satellite images is limited to classification and segmentation tasks. Extending this processing to it's next logical level, in this paper we propose a lightweight pipeline for onboard panoptic segmentation of multi-spectral satellite images. Panoptic segmentation offers major economic and environmental insights, ranging from yield estimation from agricultural lands to intelligence for complex military applications. Nevertheless, the on-board intelligence extraction raises several challenges due to the loss of temporal observations and the need to generate predictions from a single image sample. To address this challenge, we propose a multimodal teacher network based on a cross-modality attention-based fusion strategy to improve the segmentation accuracy by exploiting data from multiple modes. We also propose an online knowledge distillation framework to transfer the knowledge learned by this multi-modal teacher network to a uni-modal student which receives only a single frame input, and is more appropriate for an on-board environment. We benchmark our approach against existing state-ofthe-art panoptic segmentation models using the PASTIS multispectral panoptic segmentation dataset considering an on-board processing setting. Our evaluations demonstrate a substantial 10.7%, 11.9% and 10.6% increase in Segmentation Quality (SQ), Recognition Quality (RQ), and Panoptic Quality (PQ) metrics compared to the existing state-of-the-art model when it is evaluated in an on-board processing setting.

show abstract

Section: A Results Of the Fusion Modelmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: B Knowledge Distillation Frameworkmentioning

confidence: 99%

“…Then PQ is the product of SQ and RQ (i.e. P R = SQ × RQ), thus combining the detection, classification and delineation accuracies [29].…”

Section: Evaluationsmentioning

confidence: 99%