Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders

Abstract

Earth observation (EO) sensors deliver data at daily or weekly intervals. Most land use and land cover classification (LULC) approaches, however, are designed for cloud-free and mono-temporal observations. The increasing temporal capabilities of today's sensors enable the use of temporal, along with spectral and spatial features.Domains such as speech recognition or neural machine translation, work with inherently temporal data and, today, achieve impressive results by using sequential encoder-decoder structures. Inspired by these sequence-to-sequence models, we adapt an encoder structure with convolutional recurrent layers in order to approximate a phenological model for vegetation classes based on a temporal sequence of Sentinel 2 (S2) images. In our experiments, we visualize internal activations over a sequence of cloudy and non-cloudy images and find several recurrent cells that reduce the input activity for cloudy observations. Hence, we assume that our network has learned cloud-filtering schemes solely from input data, which could alleviate the need for tedious cloud-filtering as a preprocessing step for many EO approaches. Moreover, using unfiltered temporal series of top-of-atmosphere (TOA) reflectance data, our experiments achieved state-of-the-art classification accuracies on a large number of crop classes with minimal preprocessing, compared to other classification approaches.

Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series

“…First developed for sequential data, RNN models have been recently applied to several classification tasks in remote sensing, especially for crop mapping [45,46,48]. They share the learned features across different positions.…”

“…Following the most recent studies, we have trained bidirectional RNNs composed of the stack of three GRUs, one dense layer (256 neurons) and a Softmax layer [46,48]. The same number of neurons is used in the three GRUs.…”

A framework for large-scale mapping of human settlement extent from Sentinel-2 images via fully convolutional neural networks

“…2 illustrates the processed Sentinel-2 image of central Munich, Germany, and the reference data. There are two approaches for remote sensing image classification via deep learning: working with either patch-based CNNs designed for image classification [24,26,30,31,32,49,50,51] or encoder-decoder-like neural networks designed for semantic segmentation [25,27,28,29]. The former works under the assumption of just a single label for each image patch, and applies the trained model to the image of a study area via a sliding window approach, with the target GSD as the stride of the sliding window.…”

“…First, getting sufficient reliable pixel-wise ground truth data, a major prerequisite for deep learning-based approaches, is more challenging than labelling standard photos that are the main subject of computer vision research. Therefore, we suggest to create annotations by exploiting geo-referenced map products such as the CORINE Land Cover data [37] and the MOD500 data [38,35], as well as governmental data [30], which contains information relevant to the task one seeks to achieve. Second, remote sensing images differ significantly in appearance from the close-range images used in the standard literature on scene segmentation [39].…”

ProgNet: Covid-19 prognosis using recurrent and convolutional neural networks

“…Recurrent neural networks (RNN) are a family of deep learning methods designed to manage temporal correlations between images in time series. These networks have recurrent connections in the sense that they keep information in memory: they can take into account at time T n a number of past states T i where i < n. These networks have been used in remote sensing to assess change detection in multi-spectral and hyper-spectral images [32]. RNNs are able to memorize information for a limited time and start to "forget" after a certain number of iterations, which makes training for many applications complicated.…”