Geo-Supervised Visual Depth Prediction

Fei, Xiaohan; Wong, Alex; Soatto, Stefano

doi:10.1109/lra.2019.2896963

Cited by 61 publications

(50 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our VGG11 and VGG8 architectures following the late fusion paradigm [14,28], and our auxiliary pose network to predict relative pose between two frames for constructing our photometric and pose consistency loss (Eqn. 6,8). Our auxiliary pose network is used only in training and not inference.…”

Section: Late Fusion Vgg11mentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Depth Completion From Visual Inertial Odometry

Wong

Fei

Tsuei

et al. 2020

IEEE Robot. Autom. Lett.

Self Cite

188

View full text Add to dashboard Cite

We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visualinertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to "self-supervision," measuring photometric consistency across time, forward-backward pose consistency, and geometric compatibility with the sparse point cloud. We also launch the first visual-inertial + depth dataset, which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, and show stateof-the-art performance on it.

show abstract

Section: Late Fusion Vgg11mentioning

confidence: 99%

“…Our auxiliary pose network contains ≈ 1M parameters and is only used during training to construct the photometric and pose consistency loss (Eqn. 6,8). The output is averaged along its width and height dimensions to result in a 6 element vector -of which 3 elements are used to compose rotation (Sec.…”

Section: Appendix a Void Datasetmentioning

confidence: 99%

Unsupervised Depth Completion From Visual Inertial Odometry

Wong

Fei

Tsuei

et al. 2020

IEEE Robot. Autom. Lett.

Self Cite

188

View full text Add to dashboard Cite

show abstract

“…Since Laina [40] constructed a fully convolutional architecture to predict the depth map, following works [41], [42] benefit from the increasing ability of FCN and achieve promising results. Besides, Fei [43] proposed a semantically informed geometric loss while Wei [44] uses a virtual normal loss to constraint the structure information. Like in semantic segmentation, there are also some works try to replace the encoder with efficiency backbones [13], [44], [45] to decrease the computational cost, but suffer from the training problem limited by the ability of the compact network.…”

Section: Depth Estimationmentioning

confidence: 99%

Structured Knowledge Distillation for Semantic Segmentation

Liu

Chen

Liu

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

507

270

View full text Add to dashboard Cite

In this paper, we consider transferring the structure information from large networks to small ones for dense prediction tasks. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to small networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pairwise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by extensive experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection.

show abstract

“…Building on these observations, we choose to learn the easier tasks first, then use them as guidance to the more complex task of depth estimation through our novel consensus loss terms of Eqs. (7) and (6). See supplementary material for more details on the training procedure.…”

Section: Implementation Detailsmentioning

confidence: 99%

SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation

Ramamonjisoa

Lepetit

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

116

View full text Add to dashboard Cite

We introduce SharpNet, a method that predicts an accurate depth map for an input color image, with a particular attention to the reconstruction of occluding contours: Occluding contours are an important cue for object recognition, and for realistic integration of virtual objects in Augmented Reality, but they are also notoriously difficult to reconstruct accurately. For example, they are a challenge for stereo-based reconstruction methods, as points around an occluding contour are visible in only one image. Inspired by recent methods that introduce normal estimation to improve depth prediction, we introduce a novel term that constrains depth and occluding contours predictions. Since ground truth depth is difficult to obtain with pixel-perfect accuracy along occluding contours, we use synthetic images for training, followed by fine-tuning on real data. We demonstrate our approach on the challenging NYUv2-Depth dataset, and show that our method outperforms the state-of-the-art along occluding contours, while performing on par with the best recent methods for the rest of the images. Its accuracy along the occluding contours is actually better than the "ground truth" acquired by a depth camera based on structured light. We show this by introducing a new benchmark based on NYUv2-Depth for evaluating occluding contours in monocular reconstruction, which is our second contribution.

show abstract

Geo-Supervised Visual Depth Prediction

Cited by 61 publications

References 29 publications

Unsupervised Depth Completion From Visual Inertial Odometry

Unsupervised Depth Completion From Visual Inertial Odometry

Structured Knowledge Distillation for Semantic Segmentation

SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation

Contact Info

Product

Resources

About