Multitask Deep Neural Networks for Tele-Wide Stereo Matching

Abstract

In this paper, we propose deep learning solutions for the estimation of the real world depth of elements in a scene captured by two cameras with different field of views. We consider a realistic smartphone scenario, where the first field of view (FOV) is a wide FOV with 1× the optical zoom, and the second FOV is contained in the first FOV captured by a tele zoom lens with 2× the optical zoom. We refer to the problem of estimating the depth for all elements in the union of the FOVs which corresponds to the Wide FOV as 'tele-wide stereo matching'. Traditional approaches can only estimate the disparity or depth in the overlapped FOV, corresponding to the Tele FOV, using stereo matching algorithms. To benchmark this novel problem, we introduce a single-image inverse-depth estimation (SIDE) solution to estimate the disparity from the image corresponding to the union Wide FOV only. We also design a novel multitask tele-wide stereo matching deep neural network (MT-TW-SMNet), which is the first to combine the stereo matching and the single image depth tasks in one network. Moreover, we propose multiple methods for the fusion between the above networks. For example, we have input feature fusion, that utilizes the disparity estimated by stereo-matching as an additional input feature for SIDE. We also designed networks for decision fusion, that deploys a stacked hour glass (SHG) network for fusion and refinement of the disparity maps from both the SIDE network and the MT-TW-SMNet. These fusion schemes significantly improve the accuracy. Experimental results on KITTI and SceneFlow datasets demonstrate that our proposed approaches provide a reasonable solution to the tele-wide stereo matching problem. We demonstrate the effectiveness of our solutions in generating the Bokeh effect on the full Wide FOV.

Robust Full-Fov Depth Estimation in Tele-Wide Camera System

“…The proposed network is compared with state-of-the-art single image depth prediction methods [11,13] and a pure DNN-based tele-wide stereo matching method [14]. All of the test codes of these methods are obtained either from their official websites or from authors.…”

“…Recently Deep Neural Network (DNN) has been successfully applied to the single image depth prediction [8,9,10,11,12,13] and tele-wide stereo depth estimation [14]. Eigen et al [8] combined a coarse global prediction network based on the entire image with a refinement network, and proposed (c) Traditional tele-FoV stereo-matching method: matching cost calculation/optimization [3] + post processing [7]; (d) DNN-based single image depth method in [11]; (e) DNNbased single image depth method in [13]; (f) Pure DNN-based tele-wide stereo matching method in [14]; (g) Our result.…”

“…Li et al [13] presented a large depth dataset called MegaDepth, which is generated from collected multi-view internet photos, the hourglass model trained on this dataset shows good results on wild test images. EI-Khamy et al [14] proposed a stereo matching neural network for tele-wide camera system so as to estimate the full-FoV depth. Pure DNN-based methods usually have low robustness for scenarios which are not covered by training dataset [15,16] (as shown in Fig.…”

“…Depth comparisons. (a) Wide image; (b) Tele image; (c) Traditional tele-FoV stereo-matching method: matching cost calculation/optimization [3] + post processing [7]; (d) DNN-based single image depth method in[11]; (e) DNNbased single image depth method in[13]; (f) Pure DNN-based tele-wide stereo matching method in[14]; (g) Our result.…”