Importance-Aware Semantic Segmentation in Self-Driving with Discrete Wasserstein Training

Abstract

Semantic segmentation (SS) is an important perception manner for self-driving cars and robotics, which classifies each pixel into a pre-determined class. The widely-used cross entropy (CE) loss-based deep networks has achieved significant progress w.r.t. the mean Intersection-over Union (mIoU). However, the cross entropy loss can not take the different importance of each class in an self-driving system into account. For example, pedestrians in the image should be much more important than the surrounding buildings when make a decisions in the driving, so their segmentation results are expected to be as accurate as possible. In this paper, we propose to incorporate the importance-aware inter-class correlation in a Wasserstein training framework by configuring its ground distance matrix. The ground distance matrix can be pre-defined following a priori in a specific task, and the previous importance-ignored methods can be the particular cases. From an optimization perspective, we also extend our ground metric to a linear, convex or concave increasing function w.r.t. pre-defined ground distance. We evaluate our method on CamVid and Cityscapes datasets with different backbones (SegNet, ENet, FCN and Deeplab) in a plug and play fashion. In our extenssive experiments, Wasserstein loss demonstrates superior segmentation performance on the predefined critical classes for safe-driving.

Embedding Semantic Hierarchy in Discrete Optimal Transport for Risk Minimization

“…In this paper, we resort to the optimal transport distance as an alternative for empirical risk minimization [9,10,11,12]. With the low-cost modification of the loss function perspective, our solution can be added on any up-to-date general deep networks in a plug-and-play fashion.…”

Identity-aware Facial Expression Recognition in Compressed Video

“…Video-based FER. With the fast development of deep learning [24], [25], [26], [27], [28], [29], [30], [31], both the frame aggregation and spatiotemporal FER networks are developed and outperforms the conventional methods [32], [33]. The frame aggregation methods can utilize the image-based FER networks by making the decision-level [34] or the featurelevel frame-wise aggregation [35].…”

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

“…In contrast, our proposed framework is essentially learning-based without the need for explicit 3D reconstruction [69,77,28]. Learning-based NVS emerges with the development of convolutional neural networks (CNN) [80,52,43,55,47,46,21,48,42]. Early attempts directly map an input image to a paired target image with an encoder-decoder structure [11].…”