Unsupervised Learning of Depth and Ego-Motion From Cylindrical Panoramic Video

Sharma, Alisha; Ventura, Jonathan

doi:10.1109/aivr46125.2019.00018

Cited by 17 publications

(8 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4) We demonstrate how to use the trained neural network for novel view synthesis and stereo panorama conversion from a single input panorama. This journal paper extends our previous conference paper [16] in several ways. We evaluate our method on a larger and more complete synthetic dataset rendered using the CARLA simulator [17] that allows us to more effectively evaluate our approach.…”

Section: Introductionsupporting

confidence: 70%

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

Sharma,

Nett,

Ventura

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce a convolutional neural network model for unsupervised learning of depth and ego-motion from cylindrical panoramic video. Panoramic depth estimation is an important technology for applications such as virtual reality, 3D modeling, and autonomous robotic navigation. In contrast to previous approaches for applying convolutional neural networks to panoramic imagery, we use the cylindrical panoramic projection which allows for the use of the traditional CNN layers such as convolutional filters and max pooling without modification. Our evaluation of synthetic and real data shows that unsupervised learning of depth and ego-motion on cylindrical panoramic images can produce high-quality depth maps and that an increased field-of-view improves ego-motion estimation accuracy. We create two new datasets to evaluate our approach: a synthetic dataset created using the CARLA simulator, and Headcam, a novel dataset of panoramic video collected from a helmet-mounted camera while biking in an urban setting. We also apply our network to the problem of converting monocular panoramas to stereo panoramas.

show abstract

Section: Introductionsupporting

confidence: 70%

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

Sharma,

Nett,

Ventura

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the vertical coordinate, it projects a surface of a sphere onto a cylinder using the tangent of latitude φ, which can be envisioned by surrounding the circumference of the sphere with a flat piece of paper. Using the cylindrical projection, several authors estimate the depth from wide angle cameras [21,23]. The projection is expressed as follows:…”

Section: Cylindrical Projectionmentioning

confidence: 99%

Expandable Spherical Projection and Feature Concatenation Methods for Real-Time Road Object Detection Using Fisheye Image

Kim

Park

2022

Applied Sciences

View full text Add to dashboard Cite

Fisheye lens cameras are widely used in such applications where a large field of view (FOV) is necessary. A large FOV can provide an enhanced understanding of the surrounding environment and can be an effective solution for detecting the objects in automotive applications. However, this comes with the cost of strong radial distortions and irregular size of objects depending on the location in an image. Therefore, we propose a new fisheye image warping method called Expandable Spherical Projection to expand the center and boundary regions in which smaller objects are mostly located. The proposed method produces undistorted objects especially in the image boundary and a less unwanted background in the bounding boxes. Additionally, we propose three multi-scale feature concatenation methods and provide the analysis of the influence from the three concatenation methods in a real-time object detector. Multiple fisheye image datasets are employed to demonstrate the effectiveness of the proposed projection and feature concatenation methods. From the experimental results, we find that the proposed Expandable Spherical projection and the LCat feature concatenation yield the best AP performance, which is up to 4.7% improvement compared to the original fisheye image datasets and the baseline model.

show abstract

“…It has previously been observed that L syn suffers from a gradient locality problem, where both depth loss and L syn stagnate, since it relies on the comparison of localized pixel intensities [32,42]. Multi-scale regularization, i.e., estimating disparities and computing L syn at different spatial scales (L ms ), has been reported to mitigate the gradient locality problem [32,9,10]. Additionally, an edge-aware smoothness regularizing term L smth on the disparity map has also been included [19,35] to encourage smoothness of the estimation,…”

Section: Additional Regularization and Auxiliariesmentioning

confidence: 99%

On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation

Li,

Drenkow,

Ding

et al. 2021

Preprint

View full text Add to dashboard Cite

Scene depth estimation from stereo and monocular imagery is critical for extracting 3D information for downstream tasks such as scene understanding. Recently, learning-based methods for depth estimation have received much attention due to their high performance and flexibility in hardware choice. However, collecting ground truth data for supervised training of these algorithms is costly or outright impossible. This circumstance suggests a need for alternative learning approaches that do not require corresponding depth measurements. Indeed, self-supervised learning of depth estimation provides an increasingly popular alternative. It is based on the idea that observed frames can be synthesized from neighboring frames if accurate depth of the scene is known -or in this case, estimated. We show empirically that -contrary to common belief -improvements in image synthesis do not necessitate improvement in depth estimation. Rather, optimizing for image synthesis can result in diverging performance with respect to the main prediction objective -depth. We attribute this diverging phenomenon to aleatoric uncertainties, which originate from data. Based on our experiments on four datasets (spanning street, indoor, and medical) and five architectures (monocular and stereo), we conclude that this diverging phenomenon is independent of the dataset domain and not mitigated by commonly used regularization techniques. To underscore the importance of this finding, we include a survey of methods which use image synthesis, totaling 127 papers over the last six years. This observed divergence has not been previously reported or studied in depth, suggesting room for future improvement of self-supervised approaches which might be impacted the finding.

show abstract

Unsupervised Learning of Depth and Ego-Motion From Cylindrical Panoramic Video

Cited by 17 publications

References 27 publications

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

Expandable Spherical Projection and Feature Concatenation Methods for Real-Time Road Object Detection Using Fisheye Image

On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation

Contact Info

Product

Resources

About