Training of Convolutional Networks on Multiple Heterogeneous Datasets for Street Scene Semantic Segmentation

Abstract

We propose a convolutional network with hierarchical classifiers for per-pixel semantic segmentation, which is able to be trained on multiple, heterogeneous datasets and exploit their semantic hierarchy. Our network is the first to be simultaneously trained on three different datasets from the intelligent vehicles domain, i.e. Cityscapes, GTSDB and Mapillary Vistas, and is able to handle different semantic levelof-detail, class imbalances, and different annotation types, i.e. dense per-pixel and sparse bounding-box labels. We assess our hierarchical approach, by comparing against flat, nonhierarchical classifiers and we show improvements in mean pixel accuracy of 13.0% for Cityscapes classes and 2.4% for Vistas classes and 32.3% for GTSDB classes. Our implementation achieves inference rates of 17 fps at a resolution of 520 x 706 for 108 classes running on a GPU.

On Boosting Semantic Street Scene Segmentation with Weak Supervision

“…II-C respectively. In addition, we address the shortcomings of pseudo groundtruth generation [12] for any type of weak labels.…”

“…The benefits of the hierarchical structure [12] are twofold: 1) it solves the problem of simultaneously training with different types of supervision, by placing classes with weak labels in the subclassifiers, and 2) it solves the semantic class incompatibilities between datasets, due to the unavailability of specific semantic classes in all datasets.…”

“…The challenge when using weak supervision for per-pixel semantic segmentation lies on the different and incompatible annotation types [12]. Our method fully solves, in a consistent and uniform manner that challenge, while training on Panagiotis Meletis (p.c.meletis@tue.nl) and Gijs Dubbelman (g.dubbelman@tue.nl) are with the Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.…”

Data Selection for training Semantic Segmentation CNNs with cross-dataset weak supervision

“…Recently, multiple dataset training of convolutional networks is gaining attention [1], [2], [3], since it offers improved performance and better generalization capabilities compared to single dataset training. Multiple dataset training is especially advantageous for training semantic segmentation networks, which requires large amounts of training examples [4].…”

“…However, factors as different dataset sizes, repetitive examples (low informative value), and high annotation costs, hamper the effectiveness of multiple dataset training. These factors especially influence methods that employ weaker forms of supervision [5], [6], [7], [2].…”