Self-Supervised Learning of Depth and Ego-Motion From Videos by Alternative Training and Geometric Constraints from 3-D to 2-D

Fang, Jiaojiao; Liu, Guizhong

doi:10.1109/tcds.2022.3152241

Cited by 6 publications

(4 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the loss calculation is challenging to implement due to the absence of a groundtruth point cloud for reference. To address this issue, Fang et al [11] proposed a method that utilizes the given point cloud structure as reliable data. Nevertheless, ICP methods inherently require iterations when calculating losses, which reduces their training speeds and fails to capture detailed information.…”

Section: D Geometric Constraintsmentioning

confidence: 99%

“…Furthermore, point clouds, which encapsulate 3D information, are derived from depth maps and are becoming increasingly utilized in visual synthesis scenarios. The primary technique involves the use of the iterative closest point (ICP) algorithm [14] for estimating point cloud errors, which is crucial for generating pseudosignals [11,15]. However, the integration of optical flows integrates additional subnetworks into the utilizes self-supervised learning model, complicating the process of learning patterns within the network.…”

Section: Introductionmentioning

confidence: 99%

“…Then, Godard et al [10] integrated error algorithms, masks, networks, and training methods that became the basis for later methods. However, this view synthesis technique overlooks vital 3D geometric information, such as point clouds [11] and optical flows [7,12].…”

Section: Introductionmentioning

confidence: 99%

“…This method transforms a point cloud into an image by projecting the cloud onto six spatial planes. Furthermore, the point cloud structure, which is characterized by minimal error and comprehensive scene attributes, addresses the no-reference issue [11]. Projecting the structure of a point cloud to obtain an image enriched with 3D geometric constraints yields a reference value image.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MonoViT-3D: Self-Supervised Monocular Depth Estimation from 3D Geometric Constraints using Point Cloud Projection

Wen,

Tang,

Sun

et al. 2024

Preprint

View full text Add to dashboard Cite

Self-supervised monocular depth estimation leverages geometric constraints to create pseudolabels, eliminating the need for ground-truth data during training. In the field of 3D geometric constraints, the optical flow and iterative closest point methods are widely used to extract geometric constraints. However, the application of the existing 3D geometric constraint methods in self-supervised learning tasks increases network complexity by introducing additional submodules and iterative loops. In this paper, we propose a point cloud projection method that replaces the iterative closest point algorithm. Following depth prediction, our method utilizes the difference of 3D gaussian measure to extract 3D point cloud structures and the difference of 2D gaussian measure for the 2D structures derived from synthesized multiview surfaces. By projecting a 3D structure onto a 2D image, the errors between the target point cloud image and the synthesized views are estimated to enhance the training process of self-supervised learning. 3D geometric constraints form a point cloud optimization problem in our approach, requiring deep prediction networks with strong receptive fields for global features. Benefiting from the global receptive field of transformers, we use a vision transformer for self-supervised learning. Furthermore, we propose an encoder block to more effectively learn the patterns of 3D geometric constraints. Experiments show that we achieve strong performance on the KITTI dataset, achieving 0.095, 0.691, 4.243, and 0.171 Abs Rel, Sq Rel, RMSE, and RMSE log values for monocular depth estimation, respectively. Moreover, the model outperforms the comparison models in 3D generation tasks conducted on the Make3D dataset. All code is available at https://github.com/weentiaan/MonoViT-3D.

show abstract

Section: D Geometric Constraintsmentioning

confidence: 99%