SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection

Abstract

Freespace detection is an essential component of visual perception for self-driving cars. The recent efforts made in data-fusion convolutional neural networks (CNNs) have significantly improved semantic driving scene segmentation. Freespace can be hypothesized as a ground plane, on which the points have similar surface normals. Hence, in this paper, we first introduce a novel module, named surface normal estimator (SNE), which can infer surface normal information from dense depth/disparity images with high accuracy and efficiency. Furthermore, we propose a data-fusion CNN architecture, referred to as RoadSeg, which can extract and fuse features from both RGB images and the inferred surface normal information for accurate freespace detection. For research purposes, we publish a large-scale synthetic freespace detection dataset, named Ready-to-Drive (R2D) road dataset, collected under different illumination and weather conditions. The experimental results demonstrate that our proposed SNE module can benefit all the stateof-the-art CNNs for freespace detection, and our SNE-RoadSeg achieves the best overall performance among different datasets.

Pixel Level Segmentation Based Drivable Road Region Detection and Steering Angle Estimation Method for Autonomous Driving on Unstructured Roads

“…Chan et al [16] have proposed instance segmentation based drivable road detection method comprising of FCN and inverse perspective mapping (IPM) technology. Fan et al [17] have proposed a Road-Seg data fusion CNN architecture, which extract and fuse the features of normal surface information and RGB images for accurate road detection. Similarly, Mendes et al [18] have also explored the fully convolution neural network for road detection.…”

“…In this section, we presented a comprehensive comparison of the existing pixel level segmentation dataset for the road detection and our benchmark. As we discussed earlier the existing proposed road detection methods have been evaluated on KITTI [34], Cityscape [35], CamVid [36], R2D [17], and Synthia [40] datasets. However, the KITTI dataset consists of three subsets (i) urban marked roads, (ii) urban unmarked roads (iii) multi marked roads.…”

“…Some samples from the KITTI, Cityscape and CamVid datasets shown in Figure 10, respectively. Synthia and Ready-to-Drive (R2D) have also been used for the evaluation of the semantic segmentation networks proposed for the road detection [17,40]. However, these datasets constructed in simulation environment as shown in Figure 10.…”

Dual Attention Feature Fusion and Adaptive Context for Accurate Segmentation of Very High-Resolution Remote Sensing Images

“…With the development of very high resolution (VHR) remote sensing technology, large amounts of satellite remote sensing images with very high resolution are obtained every day [1]. Semantic segmentation is a computer vision task that predicts the semantic category for every pixel in an image, and such comprehensive image understanding is essential for many vision-based applications such as orbital remote sensing, autonomous driving [2,3], medical image analysis, and so on [4][5][6]. However, there are still lots of challenges for the task of semantic segmentation in VHR remote sensing images with complex scenes, such as poor accuracy of multi-category semantic segmentation, poor speed of multi-category semantic segmentation, and so on.…”

SCV-Stereo: Learning Stereo Matching From a Sparse Cost Volume

“…Stereo matching aims at finding correspondences between a pair of well-rectified left and right images [1]. As a fundamental computer vision and robotics task [2,3,4,5,6], it has been studied extensively for decades [7]. Traditional approaches [8,1] consist of four main steps: (i) cost computation, (ii) cost aggregation, (iii) disparity optimization, and (iv) disparity refinement [7].…”