Time-Space tradeoff in deep learning models for crop classification on satellite multi-spectral image time series

Garnot, Vivien Sainte Fare; Landrieu, Loïc; Giordano, Sébastien; Chehata, Nesrine

doi:10.48550/arxiv.1901.10503

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, recurrent neural networks [45], such as Long Short-Term Memory (LSTM) [46] or Gated Recurrent Units (GRU) [47] were commonly used in encode-decoder architectures [48] for generative prediction of words. In Earth observation, the encoder model was utilized for change detection [41,42], and land cover [20] as well as crop type identification [39,44,49,50]. To utilize both spatial and temporal features from the time series, combinations of convolutional layers with recurrent layers [43,42] and convolutional-recurrent networks [51] have been explored and comprehensively compared in [49].…”

Section: Related Workmentioning

confidence: 99%

“…In Earth observation, the encoder model was utilized for change detection [41,42], and land cover [20] as well as crop type identification [39,44,49,50]. To utilize both spatial and temporal features from the time series, combinations of convolutional layers with recurrent layers [43,42] and convolutional-recurrent networks [51] have been explored and comprehensively compared in [49]. These recurrent neural network encoders can be augmented by soft-attention mechanisms, originally developed for machine translation [52], as tested in [44].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Self-attention for raw optical Satellite Time Series Classification

Rußwurm,

Körner

2019

Preprint

View full text Add to dashboard Cite

Deep learning methods have received increasing interest by the remote sensing community for multi-temporal land cover classification in recent years. Convolutional Neural networks that elementwise compare a time series with learned kernels, and recurrent neural networks that sequentially process temporal data have dominated the state-ofthe-art in the classification of vegetation from satellite time series. In natural language processing, a third mechanism has emerged and is extensively employed for semantic information extraction from sequences of words and language modeling. This self-attention mechanism allows a neural network to selectively extract features from specific times in the input sequence thus suppressing non-classification relevant information. Today, self-attention based neural networks dominate the state-of-the-art in natural language processing but are hardly explored and tested in the remote sensing context.In this work, we embed self-attention in the canon of deep learning mechanisms for satellite time series classification for vegetation modeling and crop type identification. We compare it quantitatively to convolution, and recurrence and test four models that each exclusively relies on one of these mechanisms. The models are trained to identify the type of vegetation on crop parcels using raw and preprocessed Sentinel 2 time series over one entire year. To obtain an objective measure we find the best possible performance for each of the models by a large-scale hyperparameter search with more than 2400 validation runs. Beyond the quantitative comparison, we qualitatively analyze the models by an easy-to-implement, but yet effective feature importance analysis based on gradient back-propagation that exploits the differentiable nature of deep learning models. Finally, we look into the self-attention transformer model and visualize attention scores as bipartite graphs in the context of the input time series and a low-dimensional representation of internal hidden states using t-distributed stochastic neighborhood embedding (t-SNE).

show abstract