FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

Abstract

Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task, i.e., a sparse-to-coarse stage and a coarse-to-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-to-fine stage with coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

SemAttNet: Toward Attention-Based Semantic Aware Guided Depth Completion

“…The two branch network architectures [12]- [14], [17], [18] consist of RGB and sparse depth map branches. The RGB branch extracts color dominant information, e.g., object boundaries, which is actively fused with a sparse depth map branch at multiple stages.…”

“…The earlier two-branch methods [13], [14], [17], [18] relied heavily on the color image to extract both semantic and color-dominant information. However, color images alone might not be able to provide this information.…”

“…We evaluate the performance of our method against different state-of-the-art (SoTA) methods on KITTI depth completion [19] 743.69 209.28 2.07 0.90 NLSPN [4] 741.68 199.59 1.99 0.84 GuideNet [8] 736.24 218.83 2.25 0.99 FCFRNet [18] 735.81 217. 15 2.20 0.98 PENet [13] 730.08 210.55 2.17 0.94 RigNet [14] 713.…”

Non-learning Stereo-aided Depth Completion under Mis-projection via Selective Stereo Matching

“…In addition, NNs have been applied to depth completion tasks. The most common approach is to train networks with ground truth dense depth maps [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22]. Recently, selfsupervised and semi-supervised methods have been examined because it is difficult to acquire the dense ground truth.…”

“…An alternative to address the sparsity of LiDAR is the depth completion, and the most common approach uses a single synchronized image as a guide [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22]. These methods generate a sparse depth map by projecting LiDAR points onto the image, and then the depth map is completed using pixel intensities.…”