An Unsupervised Temporal Consistency (TC) Loss to Improve the Performance of Semantic Segmentation Networks

Varghese, Serin; Gujamagadi, Sharat; Klingner, Marvin; Kapoor, Nikhil; Bär, Andreas; Schneider, Ján; Maag, Kira; Schlicht, Peter; Hüger, Fabian; Fingscheidt, Tim

doi:10.1109/cvprw53098.2021.00010

Cited by 10 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhuang et al [144] propose a dual deep supervision (DDS), where pseudo-labels of an intermediate frame from the interval under consideration is used to calculate an additional supervisory signal in order to improve model optimization in the absence of sufficient data. Varghese et al [152] explore unlabeled video sequences in order to improve temporal consistency of single-frame methods. Pseudo-labels and optical flow-based label propagation are leveraged in a parallel stage of model training.…”

Section: Labeling Efficiencymentioning

confidence: 99%

A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design

Manfio¹,

Osório²

2023

Preprint

View full text Add to dashboard Cite

Semantic image and video segmentation stand among the most important tasks in computer vision nowadays, since they provide a complete and meaningful representation of the environment by means of a dense classification of the pixels in a given scene. Recently, Deep Learning, and more precisely Convolutional Neural Networks, have boosted semantic segmentation to a new level in terms of performance and generalization capabilities. However, designing Deep Semantic Segmentation models is a complex task, as it may involve application-dependent aspects. Particularly, when considering autonomous driving applications, the robustness-efficiency tradeoff, as well as intrinsic limitations -computational/memory bounds and data-scarcity -and constraints -real-time inference -should be taken into consideration. In this respect, the use of additional data modalities, such as depth perception for reasoning on the geometry of a scene, and temporal cues from videos to explore redundancy and consistency, are promising directions yet not explored to their full potential in the literature. In this paper, we conduct a survey on the most relevant and recent advances in Deep Semantic Segmentation in the context of vision for autonomous vehicles, from three different perspectives: efficiency-oriented model development for real-time operation, RGB-Depth data integration (RGB-D semantic segmentation), and the use of temporal information from videos in temporalaware models. Our main objective is to provide a comprehensive discussion on the main methods, advantages, limitations, results and challenges faced from each perspective, so that the reader can not only get started, but also be up to date in respect to recent advances in this exciting and challenging research field.

show abstract

Section: Labeling Efficiencymentioning

confidence: 99%

A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design

Manfio¹,

Osório²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Some approaches investigate the use of temporal information only during training to improve the temporal consistency of single-frame networks. [10] and [11] achieve this by propagating the segmentation masks in consecutive frames by optical flow.…”

Section: Related Workmentioning

confidence: 99%

Temporal Context for Robust Maritime Obstacle Detection

Žust¹,

Kristan²

2022

Preprint

View full text Add to dashboard Cite

Robust maritime obstacle detection is essential for fully autonomous unmanned surface vehicles (USVs). The currently widely adopted segmentation-based obstacle detection methods are prone to misclassification of object reflections and sun glitter as obstacles, producing many false positive detections, effectively rendering the methods impractical for USV navigation. However, water-turbulence-induced temporal appearance changes on object reflections are very distinctive from the appearance dynamics of true objects. We harness this property to design WaSR-T, a novel maritime obstacle detection network, that extracts the temporal context from a sequence of recent frames to reduce ambiguity. By learning the local temporal characteristics of object reflection on the water surface, WaSR-T substantially improves obstacle detection accuracy in the presence of reflections and glitter. Compared with existing single-frame methods, WaSR-T reduces the number of false positive detections by 41% overall and by over 53% within the danger zone of the boat, while preserving a high recall, and achieving new state-of-the-art performance on the challenging MODS maritime obstacle detection benchmark.

show abstract

“…In tasks where time is a key component, such as scene flow, a good temporal prior is essential to capture meaningful representations [39]. Notice that also pixel-wise tasks, not directly related to time evolution, can benefit from temporal consistency both during training and inference, as for example depth estimation [29], [30] and semantic segmentation [40].…”

Section: Temporal Consistency For Self-supervisionmentioning

confidence: 99%

Time-to-Label: Temporal Consistency for Self-Supervised Monocular 3D Object Detection

Mouawad,

Brasch,

Manhardt

et al. 2022

Preprint

View full text Add to dashboard Cite

Monocular 3D object detection continues to attract attention due to the cost benefits and wider availability of RGB cameras. Despite the recent advances and the ability to acquire data at scale, annotation cost and complexity still limit the size of 3D object detection datasets in the supervised settings. Self-supervised methods, on the other hand, aim at training deep networks relying on pretext tasks or various consistency constraints. Moreover, other 3D perception tasks (such as depth estimation) have shown the benefits of temporal priors as a self-supervision signal. In this work, we argue that the temporal consistency on the level of object poses, provides an important supervision signal given the strong prior on physical motion. Specifically, we propose a self-supervised loss which uses this consistency, in addition to render-andcompare losses, to refine noisy pose predictions and derive high-quality pseudo labels. To assess the effectiveness of the proposed method, we finetune a synthetically trained monocular 3D object detection model using the pseudo-labels that we generated on real data. Evaluation on the standard KITTI3D benchmark demonstrates that our method reaches competitive performance compared to other monocular self-supervised and supervised methods.

show abstract

An Unsupervised Temporal Consistency (TC) Loss to Improve the Performance of Semantic Segmentation Networks

Cited by 10 publications

References 18 publications

A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design

A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design

Temporal Context for Robust Maritime Obstacle Detection

Time-to-Label: Temporal Consistency for Self-Supervised Monocular 3D Object Detection

Contact Info

Product

Resources

About