I spy with my little eye: Learning optimal filters for cross-modal stereo under projected patterns

Abstract

With the introduction of the Kinect as a gaming interfaces, its broad commercial accessibility and high quality depth sensor has attracted the attention not only from consumers but also from researchers in the robotics community. The active sensing technique of the Kinect produces robust depth maps for reliable human pose estimation. But for a broader range of applications in robotic perception, its active sensing approach fails under many operating conditions such like objects with specular and transparent surfaces.Recently, an initial study has shown that part of the arising problems can be alleviated by complimenting the active sensing scheme with passive, cross-modal stereo between the Kinect's rgb and ir camera. However, the method is troubled by interference from the IR projector that is required for the active depth sensing method. We investigate these issues and conduct a more detailed study of the physical characteristics of the sensors as well as propose a more general method that learns optimal filters for cross-modal stereo under projected patterns. Our approach improves results over the baseline in a point-cloud-based object segmentation task without modifications of the kinect hardware and despite the interference by the projector.

Improved Depth Recovery In Consumer Depth Cameras via Disparity Space Fusion within Cross-spectral Stereo

“…Missing depth information is recovered for transparent and specular objects in addition to that missing due to inter-object occlusions and other sensing noise. This directly extends prior work [2,3] which is shown to produce lesser disparity recovery and requires computational expensive scene dependant optimization. By contrast, we offer improved depth recovery from a single sensing unit (consumer depth camera) without the need for individual per scene optimization, making it highly suitable for mobile sensing applications and dynamic scenes.…”

“…Future work will investigate the negation of structured light pattern effects within the infrared component of the process [3] and the generality of the proposed disparity space image fusion approach for use with alternative stereo matching and disparity optimization.…”

“…With reference to this prior work on cross-spectral stereo, the previously discussed pseudoinfrared driven depth improvement approach proposed by Chiu et al [2,3] most closely follows the early work of a number of authors [5,6,7,11] using simulated cross-spectral data (i.e. where one image has undergone a radiometric transform to simulate an infrared image).…”

“…By contrast to this earlier RGB-D depth improvement prior work [2,3], we propose the use of "best in class" dense gradient features from [15] to facilitate recovery of secondary cross-spectral stereo disparity directly from the depth camera {I ir , I RGB } image pair. Our main contribution is the fusion of this secondary cross-spectral (CS) disparity information with a priori depth information, obtained via conventional structured light (SL) sensing, within the disparity space image constructed prior to conventional disparity optimization for scene depth recovery.…”

“…Later work [3] replaces the scalar weights with localized 3×3 patch filters derived from prior learning but achieves only a marginal performance improvement. In both cases, a simple late-stage union of the resulting stereo depth information with that from structured light shows improved depth recovery for transparent and specular surfaces where conventional consumer depth cameras of this configuration fail [2,3].…”