Fully convolutional open set segmentation

Abstract

In traditional semantic segmentation, knowing about all existing classes is essential to yield effective results with the majority of existing approaches. However, these methods trained in a Closed Set of classes fail when new classes are found in the test phase, not being able to recognize that an unseen class has been fed. This means that they are not suitable for Open Set scenarios, which are very common in real-world computer vision and remote sensing applications. In this paper, we discuss the limitations of Closed Set segmentation and propose two fully convolutional approaches to effectively address Open Set semantic segmentation: OpenFCN and OpenPCS. OpenFCN is based on the well-known OpenMax algorithm, configuring a new application of this approach in segmentation settings. Open-PCS is a fully novel approach based on feature-space from DNN activations that serve as features for computing PCA and multi-variate gaussian likelihood in a lower dimensional space. In addition to OpenPCS and aiming to reduce the RAM memory requirements of the methodology, we also propose a slight variation of the method (OpenIPCS) that uses an iteractive version of PCA able to be trained in small batches. Experiments were conducted on the well-known ISPRS Vaihingen/Potsdam and the 2018 IEEE GRSS Data Fusion Challenge datasets. OpenFCN showed little-to-no improvement when compared to the simpler and much more time efficient SoftMax thresholding, while being some orders of magnitude slower. OpenPCS achieved promising results in almost all experiments by overcoming both OpenFCN and SoftMax thresholding. OpenPCS is also a reasonable compromise between the runtime performances of the extremely fast SoftMax thresholding and the extremely

Conditional Reconstruction for Open-set Semantic Segmentation

“…Three datasets were selected to evaluate the proposed method: Vaihingen 1 and Potsdam 1 datasets; and the Houston GRSS 2018 Data Fusion dataset [13]. As the selected datasets for this work are intended for closed set segmentation, we emulate open set environments using the Leave One Class Out (LOCO) protocol [8]. As the final model for the open set pixel recognition process, we choose the model with the best AUROC result calculated on the validation set.…”

“…All experiments were conducted using the IR-R-G-nDSM channels as input, leaving the Blue channel out. We separated images from the original training/test set used by OpenPCS [8] to serve as a validation set: area 23 for Vaihingen; and areas 3 11 and 6 14 for Potsdam. For both datasets we tested 5 different scenarios in which one class was selected as UUC for each case.…”

“…in real-world scenarios, where classes not seen during training appear on the deploy phase. In such cases, the method must be able to identify the pixels of unseen or unknown classes while still correctly segmenting known classes [6,7,8], which is the definition of Open Set Semantic Segmentation (OS3).…”

“…The first fully convolutional architectures for OS3 were proposed by Oliveira et al [8]. Open Fully Convolutional Network (OpenFCN) is a fully convolutional extension of OpenPixel [7] for dense labeling tasks.…”

“…Open Fully Convolutional Network (OpenFCN) is a fully convolutional extension of OpenPixel [7] for dense labeling tasks. Open Principal Component Scoring (OpenPCS) [8] uses the intermediate feature activations from closed set networks to fit gaussian distributions in a low-dimensional projection, obtaining a likelihood score for OOD pixel detection.…”