Unsupervised Learning of Depth, Optical Flow and Pose With Occlusion From 3D Geometry

Wang, Guangming; Zhang, Chi; Wang, Hesheng; Wang, Jingchuan; Wang, Yong; Wang, Xinlei

doi:10.1109/tits.2020.3010418

Cited by 48 publications

(18 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On both monocular depth estimation and monocular visual odometry by the unsupervised learning of depth and pose, our method outperforms recent state-of-the-art methods. Besides, our method does not need auxiliary tasks, such as optical flow estimation [9]- [12], [26], [39], semantic segmentation [39] or dynamic mask estimation [11], [26], normal estimation [8]. This paper realizes end-to-end iterative view synthesis and pose refinement to jointly optimize the pose and depth estimation networks, allowing the overall parameters to selflearn according to the optimization objective.…”

Section: B Evaluation Resultsmentioning

confidence: 99%

“…parameters from the video to make the depth prediction more accurate, and use the predicted depth map directly to deal with occlusion. DOP [12] divides an image into three regions, static regions, dynamic regions, and occlusion regions by the information of adjacent frames. They solve the occlusion problem by explicit geometric calculation using the predicted point cloud.…”

Section: Self-supervised From Monocular Videomentioning

confidence: 99%

“…The edge-aware depth smoothness loss is used to make the estimated depth smoother inside an object but can have unsmoothness at the edge of the object. The revised depth smooth loss [12] is used to mitigate depth degradation in the training process:…”

Section: B 3d Hierarchical Refinement With Multi-scale Depths and Mul...mentioning

confidence: 99%

See 2 more Smart Citations

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Depth and ego-motion estimations are essential for the localization and navigation of autonomous robots and autonomous driving. Recent studies make it possible to learn the per-pixel depth and ego-motion from the unlabeled monocular video. A novel unsupervised training framework is proposed with 3D hierarchical refinement and augmentation using explicit 3D geometry. In this framework, the depth and pose estimations are hierarchically and mutually coupled to refine the estimated pose layer by layer. The intermediate view image is proposed and synthesized by warping the pixels in an image with the estimated depth and coarse pose. Then, the residual pose transformation can be estimated from the new view image and the image of the adjacent frame to refine the coarse pose. The iterative refinement is implemented in a differentiable manner in this paper, making the whole framework optimized uniformly. Meanwhile, a new image augmentation method is proposed for the pose estimation by synthesizing a new view image, which creatively augments the pose in 3D space but gets a new augmented 2D image. The experiments on KITTI demonstrate that our depth estimation achieves state-of-the-art performance and even surpasses recent approaches that utilize other auxiliary tasks. Our visual odometry outperforms all recent unsupervised monocular learning-based methods and achieves competitive performance to the geometrybased method, ORB-SLAM2 with back-end optimization.

show abstract

Section: B Evaluation Resultsmentioning

confidence: 99%

Section: Self-supervised From Monocular Videomentioning

confidence: 99%

See 1 more Smart Citation

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Works that employ these algorithmic networks are trained with images and, in the case of supervised learning, with ground truth depth data. The trained network can then be used to estimate depths from unknown 2D images, so called monocular-or single-view depth [7][8][9][10][11].…”

Section: Introductionmentioning

confidence: 99%

Depth from axial differential perspective

2022

View full text Add to dashboard Cite

We introduce an imaging-based passive on-axis technique for measuring the distance of individual objects in complex scenes. Two axially separated pupil positions acquire images (can be realized simultaneously or sequentially). Based on the difference in magnification for objects within the images, the distance to the objects can be inferred. The method avoids some of the disadvantages of passive triangulation sensors (e.g., correspondence, shadowing), is easy to implement and offers high lateral resolution. Due to the principle of operation it is especially suited for applications requiring only low to medium axial resolution. Theoretical findings, as well as follow-up experimental measurements, show obtainable resolutions in the range of few centimeters for distances of up to several meters.

show abstract

“…Recently, some works (Liu et al, 2019a;Wang et al, 2021d;Puy et al, 2020;Li et al, 2021b,a;Wang et al, 2021a) have been done to realize supervised estimation of 3D scene flow from two consecutive frames of point clouds. However, just like it is difficult to obtain the ground truth of optical flow (Wang et al, 2021c(Wang et al, , 2020b, the ground truth of 3D scene flow is also difficult to obtain. Therefore, it is essential to perform unsupervised learning of 3D scene flow.…”

Section: Introductionmentioning

confidence: 99%

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Wang

Chaokang

Shen

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

3D scene flow presents the 3D motion of each point in the 3D space, which forms the fundamental 3D motion perception for autonomous driving and server robots. Although the RGBD camera or LiDAR capture discrete 3D points in space, the objects and motions usually are continuous in the macro world. That is, the objects keep themselves consistent as they flow from the current frame to the next frame. Based on this insight, the Generative Adversarial Networks (GAN) is utilized to self-learn 3D scene flow with no need for ground truth. The fake point cloud of the second frame is synthesized from the predicted scene flow and the point cloud of the first frame. The adversarial training of the generator and discriminator is realized through synthesizing indistinguishable fake point cloud and discriminating the real point cloud and the synthesized fake point cloud. The experiments on KITTI scene flow dataset show that our method realizes promising results without ground truth. Just like a human observing a real-world scene, the proposed approach is capable of determining the consistency of the scene at different moments in spite of the exact flow value of each point is unknown in advance. Corresponding author(s) Email: wanghesheng@sjtu.edu.cn

show abstract

Unsupervised Learning of Depth, Optical Flow and Pose With Occlusion From 3D Geometry

Cited by 48 publications

References 53 publications

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Depth from axial differential perspective

SFGAN: Unsupervised Generative Adversarial Learning of 3D Scene Flow from the 3D Scene Self

Contact Info

Product

Resources

About