Exploring RGB+Depth Fusion for Real-Time Object Detection

Abstract

In this paper, we investigate whether fusing depth information on top of normal RGB data for camera-based object detection can help to increase the performance of current state-of-the-art single-shot detection networks. Indeed, depth sensing is easily acquired using depth cameras such as a Kinect or stereo setups. We investigate the optimal manner to perform this sensor fusion with a special focus on lightweight single-pass convolutional neural network (CNN) architectures, enabling real-time processing on limited hardware. For this, we implement a network architecture allowing us to parameterize at which network layer both information sources are fused together. We performed exhaustive experiments to determine the optimal fusion point in the network, from which we can conclude that fusing towards the mid to late layers provides the best results. Our best fusion models significantly outperform the baseline RGB network in both accuracy and localization of the detections.

ISETAuto: Detecting Vehicles With Depth and Radiance Information

“…These groups have used a variety of neural network architectures. Some authors input explicit depth and imaging data on independent channels that are processed separately through several network layers [4], [18], [29], [34]. After some processing, the depth and radiance channels are fused.…”

“…This design raises the question of which layer is best for merging the independent channels; one might expect that the answer depends on both the network and the data. Using a YOLOv2 network, [29] explored how variations in the merged layer influenced performance. The portions of their analysis most relevant to our work detected vehicles using data obtained from the KITTI database.…”

“…In prior work, authors developed deep learning architectures to combine radiance and depth information [4], [18], [29], [34]. The literature includes numerous innovative approaches for combining depth maps or point clouds with radiance data ( Figure 12A-C).…”

Concept Towards Segmenting Arm Areas for Robot-Based Dermatological In Vivo Measurements

“…Further, the authors introduce the 3D IoU with Volume of Overlap and Volume of Union for a better 3D bounding box proposal. Ophoff et al [44] propose a different approach. They use a separate network stream for the RGB and depth information each and fuse them by a concatenation layer.…”

Asymmetric Adaptive Fusion in a Two-Stream Network for RGB-D Human Detection

“…Some detection methods using RGB-D two-stream network have achieved a good result. Ophoff et al [ 29 ] explored the best fusion position of RGB and depth information in the CNN, from which they concluded that the best results can be obtained by feature fusion towards the mid to late layers. Gupta et al [ 30 ] proposed a depth map encoding method called HHA(horizontal disparity, height above ground, and angle with respect to gravity), which can encode the depth map into a three-channel image like RGB images.…”