Learning Stereo from Single Images

Watson, Jamie; Aodha, Oisin Mac; Turmukhambetov, Daniyar; Brostow, Gabriel J.; Firman, Michael

doi:10.1007/978-3-030-58452-8_42

Cited by 53 publications

(32 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the data gap is inevitable from the virtual data to the real autonomous driving scenario. Therefore, we propose a 6-DoF pose augmentation by generating a random pose T aug and then forward warping [38] the origin image X t to synthesizing a new arbitrary view X t aug :…”

Section: -Dof Pose Augmentation By Means Of Forward Warpingmentioning

confidence: 99%

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Depth and ego-motion estimations are essential for the localization and navigation of autonomous robots and autonomous driving. Recent studies make it possible to learn the per-pixel depth and ego-motion from the unlabeled monocular video. A novel unsupervised training framework is proposed with 3D hierarchical refinement and augmentation using explicit 3D geometry. In this framework, the depth and pose estimations are hierarchically and mutually coupled to refine the estimated pose layer by layer. The intermediate view image is proposed and synthesized by warping the pixels in an image with the estimated depth and coarse pose. Then, the residual pose transformation can be estimated from the new view image and the image of the adjacent frame to refine the coarse pose. The iterative refinement is implemented in a differentiable manner in this paper, making the whole framework optimized uniformly. Meanwhile, a new image augmentation method is proposed for the pose estimation by synthesizing a new view image, which creatively augments the pose in 3D space but gets a new augmented 2D image. The experiments on KITTI demonstrate that our depth estimation achieves state-of-the-art performance and even surpasses recent approaches that utilize other auxiliary tasks. Our visual odometry outperforms all recent unsupervised monocular learning-based methods and achieves competitive performance to the geometrybased method, ORB-SLAM2 with back-end optimization.

show abstract

Section: -Dof Pose Augmentation By Means Of Forward Warpingmentioning

confidence: 99%

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Wang¹,

Jiquan²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Strictly re- lated to our work is [61], estimating depth from single images to synthesize virtual right views and thus obtain stereo pairs, used to train deep stereo networks. Despite the analogy of using single image depth estimation, we point out that our goal differs from [61] since we aim at modeling arbitrary motions in the scene (i.e., optical flow) rather than a horizontal pixel displacement between synchronized images (i.e., disparity). Purposely, we will describe the additional strategies required to attain, from single still images, the best training data for optical flow networks.…”

Section: Related Workmentioning

confidence: 99%

“…In case the network estimates inverse depth, we bring it to the depth domain first. D 0 usually shows blurred edges [61,49], causing flying pixels in the 3D space that can be easily sharpened via edge-preserving filters [32].…”

Section: Depthstillation Pipelinementioning

confidence: 99%

“…Forward warping suffers from two well-known problems [61], that are collisions (i.e., multiple pixels from I 0 being warped to the same location in I 1 ) and holes (i.e., pixels in I 1 over which no pixel from I 0 is projected). To handle collisions, we keep track of pixels p 1 having multiple projections p 0 in a binary collision mask M (i.e., collisions are labeled as 1, other pixels as 0) and select, for each, the one having minimum depth according to camera in position c 1 , i.e.…”

Section: Depthstillation Pipelinementioning

confidence: 99%

“…A notable example is represented by the KITTI optical flow benchmarks [10,35], over which deep networks that have been trained only on synthetic data perform poorly, as witnessed by recent works [20,51,53]. This problem is known in literature and has been faced for other tasks such as semantic segmentation [15,39,43,55] or stereo depth estimation [56,57,68,61]. To fully restore a level of accuracy comparable to the one achieved on synthetic data, fine-tuning on imagery similar to the testing domain is usually required.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning optical flow from still images

Aleotti¹,

Poggi²,

Mattoccia³

2021

Preprint

View full text Add to dashboard Cite

This paper deals with the scarcity of data for training optical flow networks, highlighting the limitations of existing sources such as labeled synthetic datasets or unlabeled real videos. Specifically, we introduce a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture. Given an image, we use an off-the-shelf monocular depth estimation network to build a plausible point cloud for the observed scene. Then, we virtually move the camera in the reconstructed environment with known motion vectors and rotation angles, allowing us to synthesize both a novel view and the corresponding optical flow field connecting each pixel in the input image to the one in the new frame. When trained with our data, state-of-the-art optical flow networks achieve superior generalization to unseen real data compared to the same models trained either on annotated synthetic datasets or unlabeled videos, and better specialization if combined with synthetic images.

show abstract

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Tosi

Poggi

Benincasa

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Confidence measures for stereo gained popularity in recent years due to their improved capability to detect outliers and the increasing number of applications exploiting these cues. In this field, convolutional neural networks achieved top-performance compared to other known techniques in the literature by processing local information to tell disparity assignments from outliers. Despite this outstanding achievements, all approaches rely on clues extracted with small receptive fields thus ignoring most of the overall image content. Therefore, in this paper, we propose to exploit nearby and farther clues available from image and disparity domains to obtain a more accurate confidence estimation. While local information is very effective for detecting high frequency patterns, it lacks insights from farther regions in the scene. On the other hand, enlarging the receptive field allows to include clues from farther regions but produces smoother uncertainty estimation, not particularly accurate when dealing with high frequency patterns. For these reasons, we propose in this paper a multi-stage cascaded network to combine the best of the two worlds. Extensive experiments on three datasets using three popular stereo algorithms prove that the proposed framework outperforms state-of-the-art confidence estimation techniques.

show abstract

Learning Stereo from Single Images

Cited by 53 publications

References 75 publications

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video

Learning optical flow from still images

Beyond Local Reasoning for Stereo Confidence Estimation with Deep Learning

Contact Info

Product

Resources

About