Driving Scene Perception Network: Real-Time Joint Detection, Depth Estimation and Semantic Segmentation

Chen, Liangfu; Zeng, Yang; Ma, Jianjun; Luo, Zheng

doi:10.1109/wacv.2018.00145

Cited by 49 publications

(28 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each task has only 2 specific parametric layers, while everything else is shared in computer vision have been following this strategy; in particular, Eigen & Fergus [9] trained a single architecture (but with different copies) to predict depth, surface normals and semantic segmentation, Kokkinos [10] proposed a universal network to tackle 7 different vision tasks, Dvornik et al [11] found it beneficial to do joint semantic segmentation and object detection, while Kendall et al [12] learned optimal weights to perform instance segmentation, semantic segmentation and depth estimation all at once. Chen et al [13] built a single network with the ResNet-50 [14] backbone performing joint semantic segmentation, depth estimation and object detection. To alleviate the problem of imbalanced annotations, Kokkinos [10] chose to accumulate the gradients for each task until a certain number of examples per task is seen, while Dvornik et al [11] simply resorted to keeping the branch with no ground truth available intact until at least one example of that modality is seen.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

Nekrasov

Dharmasiri

Spek

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

119

View full text Add to dashboard Cite

Deployment of deep learning models in robotics as sensory information extractors can be a daunting task to handle, even using generic GPU cards. Here, we address three of its most prominent hurdles, namely, i) the adaptation of a single model to perform multiple tasks at once (in this work, we consider depth estimation and semantic segmentation crucial for acquiring geometric and semantic understanding of the scene), while ii) doing it in real-time, and iii) using asymmetric datasets with uneven numbers of annotations per each modality. To overcome the first two issues, we adapt a recently proposed real-time semantic segmentation network, making changes to further reduce the number of floating point operations. To approach the third issue, we embrace a simple solution based on hard knowledge distillation under the assumption of having access to a powerful 'teacher' network. We showcase how our system can be easily extended to handle more tasks, and more datasets, all at once, performing depth estimation and segmentation both indoors and outdoors with a single model. Quantitatively, we achieve results equivalent to (or better than) current state-of-the-art approaches with one forward pass costing just 13ms and 6.5 GFLOPs on 640×480 inputs. This efficiency allows us to directly incorporate the raw predictions of our network into the SemanticFusion framework [1] for dense 3D semantic reconstruction of the scene. 3

show abstract

Section: Related Workmentioning

confidence: 99%

Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

Nekrasov

Dharmasiri

Spek

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

119

View full text Add to dashboard Cite

show abstract

“…Multi-task learning (Kokkinos, 2017), (Chen et al, 2018b), (Neven et al, 2017) has been gaining significant popularity over the past few years as it has proven to be very efficient for embedded deployment. Multiple tasks like object detection, semantic segmentation, depth estimation etc can be solved simultaneously using a single model.…”

Section: Multi-task Learningmentioning

confidence: 99%

AuxNet: Auxiliary Tasks Enhanced Semantic Segmentation for Automated Driving

Chennupati

Sistu

Yogamani

et al. 2019

Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application

View full text Add to dashboard Cite

Decision making in automated driving is highly specific to the environment and thus semantic segmentation plays a key role in recognizing the objects in the environment around the car. Pixel level classification once considered a challenging task which is now becoming mature to be productized in a car. However, semantic annotation is time consuming and quite expensive. Synthetic datasets with domain adaptation techniques have been used to alleviate the lack of large annotated datasets. In this work, we explore an alternate approach of leveraging the annotations of other tasks to improve semantic segmentation. Recently, multi-task learning became a popular paradigm in automated driving which demonstrates joint learning of multiple tasks improves overall performance of each tasks. Motivated by this, we use auxiliary tasks like depth estimation to improve the performance of semantic segmentation task. We propose adaptive task loss weighting techniques to address scale issues in multi-task loss functions which become more crucial in auxiliary tasks. We experimented on automotive datasets including SYNTHIA and KITTI and obtained 3% and 5% improvement in accuracy respectively.1. Construction of auxiliary task learning architecture for semantic segmentation.2. Novel loss function weighting strategy for one main task and one auxiliary task.

show abstract

“…MultiNet [26] combines classification, detection and segmentation in a single architecture, which is based on ResNet [28] and consists of three shared encoding layers followed by task independent decoding layers. Reference [29] introduces an efficient approach for simultaneous object detection, depth estimation and pixel-level semantic segmentation using a shared convolutional architecture. Reference [27] proposes a unified neural network to detect drivable areas, lane lines, and traffic objects simultaneously, in which the three tasks are most important for autonomous driving.…”

Section: Introdctionmentioning

confidence: 99%