FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

Abstract

Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a twostage learning task, i.e., a sparse-to-coarse stage and a coarseto-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-tofine stage with coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

RigNet: Repetitive Image Guided Network for Depth Completion

“…DSPN [47] proposes a deformable SPN to adaptively generates different receptive field and affinity matrix at each pixel for effective propagation. Another pattern is using two independent branches to extract the color image and depth features respectively and then fuse them at multi-scale stages [44,48,42,25,28,16]. For example, PENet [16] employs feature addition to guide depth learning at different stages.…”

“…For example, PENet [16] employs feature addition to guide depth learning at different stages. FCFRNet [28] proposes channel-shuffle technology to enhance RGB-D feature fusion. ACMNet [54] chooses to adopt the graph propagation to capture the observed spatial contexts in the encoder stage.…”

“…NAS-FPN [12] and BiFPN [41] conduct repetitive blocks to sufficiently encode discriminative image semantics for object detection. FCFRNet [28] The former consists of a repetitive hourglass network and the latter has the same structure as RHN1. In the depth branch, we perform our novel repetitive guidance module (elaborated in Fig.…”

“…Each image is downsized to 320 × 240, and then 304 × 228 center-cropping is applied. (c) FCFRNet [28], and PENet [16]. Given sparse depth maps and the aligned color images (1st column), depth completion models output dense depth predictions (e.g., 2nd column).…”

“…Recently, multi-modal information from various sensors is involved to help generate dependable depth results, such as color images [32,2], surface normals [52,37], confidence maps [9,44], and even binaural echoes [11,34]. Particularly, the latest image guided methods [54,28,16,42] prin- cipally concentrate on using color images to guide the recovery of dense depth maps, achieving outstanding performance. However, due to the challenging environments and limited depth measurements, it's difficult for existing image guided methods to produce clear image guidance and structure-detailed depth features.…”