Stereo and ToF data fusion by learning from synthetic data

Abstract

Time-of-Flight (ToF) sensors and stereo vision systems are both capable of acquiring depth information but they have complementary characteristics and issues. A more accurate representation of the scene geometry can be obtained by fusing the two depth sources. In this paper we present a novel framework for data fusion where the contribution of the two depth sources is controlled by confidence measures that are jointly estimated using a Convolutional Neural Network. The two depth sources are fused enforcing the local consistency of depth data, taking into account the estimated confidence information. The deep network is trained using a synthetic dataset and we show how the classifier is able to generalize to different data, obtaining reliable estimations not only on synthetic data but also on real world scenes. Experimental results show that the proposed approach increases the accuracy of the depth estimation on both synthetic and real data and that it is able to outperform state-of-the-art methods.

FloatingFusion: Depth from ToF and Image-stabilized Stereo Cameras

“…ToF and RGB Fusion. Existing data-driven approaches [1,2,36] heavily rely on synthetic data, creating a domain gap. This is exacerbated when using imperfect low-power sensors such on mobile phones.…”

“…In addition, current stereo-ToF fusion [14,10,15] typically estimates disparity from stereo and ToF separately before fusion. One approach is to estimate stereo and ToF confidence to merge the disparity maps [33,1,2,37]. In contrast, our ToF estimates are directly incorporated into our disparity pipeline before depth selection.…”

“…For fusion, we need to accurately know the geometric poses of all sensors and lenses in the camera system. This leads to our second challenge: (2) As RGB spatial resolution has increased, smartphones now use optical image stabilization [40,50]: a floating lens compensates for camera body motion to avoid motion blur during exposure. Two low-power actuators suspend the lens body vertically and horizontally to provide a few degrees of in-plane rotation or translation, similar to how a third actuator translates the lens along the optical axis for focus.…”

Distance Error Correction in Time-of-Flight Cameras Using Asynchronous Integration Time

“…Light scattering errors [5], multipath errors [6], and object boundary ambiguity [7] have been identified as non-systematic errors. Compression may also result in distortion [8]. Figure 1 represents the problems in depth maps captured using ToF depth cameras.…”

“…Sensors 2020, 20,1156 2 of 17 systematic errors. Compression may also result in distortion [8]. Figure 1 represents the problems in depth maps captured using ToF depth cameras.…”