Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks

Garnot, Vivien Sainte Fare; Landrieu, Loïc

doi:10.1109/iccv48922.2021.00483

Cited by 64 publications

(104 citation statements)

References 32 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…A. Datasets 1) Panoptic Agricultural Satellite TImeSeries (PASTIS) [10] dataset: The PASTIS dataset is composed of 2,433 multispectral image patches, each with 10 spectral bands and of size 128 × 128 pixels, at 10m/pixel resolution. To the best of our knowledge, this is the only publicly available multispectral satellite image dataset with panoptic annotation.…”

Section: Methodsmentioning

confidence: 99%

“…In [10] the authors have observed that even after filtering out predominantly cloudy acquisitions, there still exists partially or completely obstructed patches. Furthermore, they highlight the utility of time-series data for resolving those obstructions and boundary ambiguities due to clouds and other atmospheric noise.…”

Section: Methodsmentioning

confidence: 99%

“…The multiple temporal observations can be used to eradicate these ambiguities in ground-based processing, however, in an on-board setting an AI algorithm does not have access to such historical data. In particular, when benchmarking the algorithms presented in [10] in a nontemporal (single frame) setting we illustrate how state-of-the-art methods struggle to generate accurate predictions (See Sec. IV-B).…”

Section: Methodsmentioning

confidence: 99%

“…1) A Teacher Backbone with Cross-Modality Attention: In this section we illustrate how the proposed cross-modality attention backbone is formulated. The approach takes inspiration from the U-TAE backbone from [10], though we incorporate specific augmentations to increase its capabilities in a multimodal setting. The spatio-temporal encoding structure is based on [10], however, prior to the decoding phase we employ cross-modality attention to extract task specific, salient information that compliments each modality.…”

Section: B Knowledge Distillation Frameworkmentioning

confidence: 99%

“…Specifically, our contributions are three-fold. First, we benchmark the performance of several state-of-the-art panoptic segmentation models in the on-board processing setting using the Panoptic Agricultural Satellite TImeSeries (PASTIS) [10] dataset. To the best of our knowledge, this is the only publicly available multi-spectral dataset with panoptic annotation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Towards On-Board Panoptic Segmentation of Multispectral Satellite Images

Fernando¹,

Fookes²,

Gammulle³

et al. 2022

Preprint

View full text Add to dashboard Cite

With tremendous advancements in low-power embedded computing devices and remote sensing instruments, the traditional satellite image processing pipeline which includes an expensive data transfer step prior to processing data on the ground is being replaced by on-board processing of captured data. This paradigm shift enables critical and time-sensitive analytic intelligence to be acquired in a timely manner onboard the satellite itself. However, at present, the on-board processing of multi-spectral satellite images is limited to classification and segmentation tasks. Extending this processing to it's next logical level, in this paper we propose a lightweight pipeline for onboard panoptic segmentation of multi-spectral satellite images. Panoptic segmentation offers major economic and environmental insights, ranging from yield estimation from agricultural lands to intelligence for complex military applications. Nevertheless, the on-board intelligence extraction raises several challenges due to the loss of temporal observations and the need to generate predictions from a single image sample. To address this challenge, we propose a multimodal teacher network based on a cross-modality attention-based fusion strategy to improve the segmentation accuracy by exploiting data from multiple modes. We also propose an online knowledge distillation framework to transfer the knowledge learned by this multi-modal teacher network to a uni-modal student which receives only a single frame input, and is more appropriate for an on-board environment. We benchmark our approach against existing state-ofthe-art panoptic segmentation models using the PASTIS multispectral panoptic segmentation dataset considering an on-board processing setting. Our evaluations demonstrate a substantial 10.7%, 11.9% and 10.6% increase in Segmentation Quality (SQ), Recognition Quality (RQ), and Panoptic Quality (PQ) metrics compared to the existing state-of-the-art model when it is evaluated in an on-board processing setting.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: B Knowledge Distillation Frameworkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations