Real-Time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-Driving Images

Sun, Lei; Yang, Kailun; Hu, Xinxin; Hu, Weijian; Wang, Kaiwei

doi:10.1109/lra.2020.3007457

Cited by 121 publications

(42 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RGB-D camera has been a rising technology in recent years [ 20 , 21 ]. Kinect v2, a common RGB-D camera, can simultaneously record the vibration video of objects in both RGB mode and depth mode.…”

Section: Methodsmentioning

confidence: 99%

Low Frequency Vibration Visual Monitoring System Based on Multi-Modal 3DCNN-ConvLSTM

Alimasi

Liu

Lyu

2020

Sensors

View full text Add to dashboard Cite

Low frequency vibration monitoring has significant implications on environmental safety and engineering practices. Vibration expressed by visual information should contain sufficient spatial information. RGB-D camera could record diverse spatial information of vibration in frame images. Deep learning can adaptively transform frame images into deep abstract features through nonlinear mapping, which is an effective method to improve the intelligence of vibration monitoring. In this paper, a multi-modal low frequency visual vibration monitoring system based on Kinect v2 and 3DCNN-ConvLSTM is proposed. Microsoft Kinect v2 collects RGB and depth video information of vibrating objects in unstable ambient light. The 3DCNN-ConvLSTM architecture can effectively learn the spatial-temporal characteristics of muti-frequency vibration. The short-term spatiotemporal feature of the collected vibration information is learned through 3D convolution networks and the long-term spatiotemporal feature is learned through convolutional LSTM. Multi-modal fusion of RGB and depth mode is used to further improve the monitoring accuracy to 93% in the low frequency vibration range of 0–10 Hz. The results show that the system can monitor low frequency vibration and meet the basic measurement requirements.

show abstract

Section: Methodsmentioning

confidence: 99%

Low Frequency Vibration Visual Monitoring System Based on Multi-Modal 3DCNN-ConvLSTM

Alimasi

Liu

Lyu

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…These groups have used a variety of neural network architectures. Some authors input explicit depth and imaging data on independent channels that are processed separately through several network layers [4], [18], [29], [34]. After some processing, the depth and radiance channels are fused.…”

Section: Related Workmentioning

confidence: 99%

“…The task we analyze is detecting 2D bounding boxes, and for this goal the depth map format improves performance significantly. Future experiments should explore how the [34] effectiveness of the RGD representation for detecting 3D bounding boxes.…”

Section: B Depth and Radiance Representationsmentioning

confidence: 99%

See 1 more Smart Citation

ISETAuto: Detecting Vehicles With Depth and Radiance Information

2021

View full text Add to dashboard Cite

show abstract

“…In order to solve this problem, a multitude of scholars have constructed real-time semantic segmentation models. Sun [ 13 ] proposed a real-time fusion semantic segmentation network called RFNet, which can effectively use complementary cross-mode information to conduct real-time RGB-D fusion semantic segmentation research, enriching the unforeseen hazard identification in real scenes. Zhao [ 14 ] proposed an image cascade network (ICNet) based on the pyramid scene analysis network, which integrates medium- and high-resolution features, while taking into account the segmentation accuracy, and uses the cascade strategy to accelerate the realization of real-time image semantic segmentation.…”

Section: Introductionmentioning

confidence: 99%

Implementation of a Lightweight Semantic Segmentation Algorithm in Road Obstacle Detection

Liu

et al. 2020

Sensors

View full text Add to dashboard Cite

Due to deep learning’s accurate cognition of the street environment, the convolutional neural network has achieved dramatic development in the application of street scenes. Considering the needs of autonomous driving and assisted driving, in a general way, computer vision technology is used to find obstacles to avoid collisions, which has made semantic segmentation a research priority in recent years. However, semantic segmentation has been constantly facing new challenges for quite a long time. Complex network depth information, large datasets, real-time requirements, etc., are typical problems that need to be solved urgently in the realization of autonomous driving technology. In order to address these problems, we propose an improved lightweight real-time semantic segmentation network, which is based on an efficient image cascading network (ICNet) architecture, using multi-scale branches and a cascaded feature fusion unit to extract rich multi-level features. In this paper, a spatial information network is designed to transmit more prior knowledge of spatial location and edge information. During the course of the training phase, we append an external loss function to enhance the learning process of the deep learning network system as well. This lightweight network can quickly perceive obstacles and detect roads in the drivable area from images to satisfy autonomous driving characteristics. The proposed model shows substantial performance on the Cityscapes dataset. With the premise of ensuring real-time performance, several sets of experimental comparisons illustrate that SP-ICNet enhances the accuracy of road obstacle detection and provides nearly ideal prediction outputs. Compared to the current popular semantic segmentation network, this study also demonstrates the effectiveness of our lightweight network for road obstacle detection in autonomous driving.

show abstract

Real-Time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-Driving Images

Cited by 121 publications

References 28 publications

Low Frequency Vibration Visual Monitoring System Based on Multi-Modal 3DCNN-ConvLSTM

Low Frequency Vibration Visual Monitoring System Based on Multi-Modal 3DCNN-ConvLSTM

ISETAuto: Detecting Vehicles With Depth and Radiance Information

Implementation of a Lightweight Semantic Segmentation Algorithm in Road Obstacle Detection

Contact Info

Product

Resources

About