Real-Time Panoramic Depth Maps from Omni-directional Stereo Images for 6 DoF Videos in Virtual Reality

Lai, Po Kong; Xie, Shuang; Lang, Jochen; Laqaruere, Robert

doi:10.1109/vr.2019.8798016

Cited by 34 publications

(24 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is equipped with 16 high-definition cameras, and is equipped with powerful processor and memory array. Each interval of two cameras constitutes a binocular stereo vision, which is responsible for the 45° field of view in the circumferential direction, and then realizes the 360° panoramic stereo imaging in the circumferential direction [ 12 ]. The current panoramic stereo imaging system still has many shortcomings, such as panoramic imaging visual effect flattening, immersion experience feeling is not strong, image processing real-time, unable to show real-time dynamic changes, image stereo matching accuracy is not high, the observer is prone to dizziness and so on [ 13 , 14 ].…”

Section: Related Workmentioning

confidence: 99%

Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision

Wang

Zhang

2021

Sensors

View full text Add to dashboard Cite

With the rapid development of the virtual reality industry, one of the bottlenecks is the scarcity of video resources. How to capture high-definition panoramic video with depth information and real-time stereo display has become a key technical problem to be solved. In this paper, the optical optimization design scheme of panoramic imaging based on binocular stereo vision is proposed. Combined with the real-time processing algorithm of multi detector mosaic panoramic stereo imaging image, a panoramic stereo real-time imaging system is developed. Firstly, the optical optimization design scheme of panoramic imaging based on binocular stereo vision is proposed, and the space coordinate calibration platform of ultra-high precision panoramic camera based on theodolite angle compensation function is constructed. The projection matrix of adjacent cameras is obtained by solving the imaging principle of binocular stereo vision. Then, a real-time registration algorithm of multi-detector mosaic image and Lucas-Kanade optical flow method based on image segmentation are proposed to realize stereo matching and depth information estimation of panoramic imaging, and the estimation results are analyzed effectively. Experimental results show that the stereo matching time of panoramic imaging is 30 ms, the registration accuracy is 0.1 pixel, the edge information of depth map is clearer, and it can meet the imaging requirements of different lighting conditions.

show abstract

Section: Related Workmentioning

confidence: 99%

Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision

Wang

Zhang

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…approaches [10,11,18,30]. While closed meshes do not introduce holes, visible distortions often appear at depth discontinuities.…”

Section: Related Workmentioning

confidence: 99%

Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video

Mühlhausen

Kappel

Kassubeck

et al. 2020

26th ACM Symposium on Virtual Reality Software and Technology

View full text Add to dashboard Cite

We present a new pipeline to enable head-motion parallax in omnidirectional stereo (ODS) panorama video rendering using a neural depth decoder. While recent ODS panorama cameras record short-baseline horizontal stereo parallax to offer the impression of binocular depth, they do not support the necessary translational degrees-of-freedom (DoF) to also provide for head-motion parallax in virtual reality (VR) applications. To overcome this limitation, we propose a pipeline that enhances the classical ODS panorama format with 6 DoF free-viewpoint rendering by decomposing the scene into a multi-layer mesh representation. Given a spherical stereo panorama video, we use the horizontal disparity to store explicit depth information for both eyes in a simple neural decoder architecture. While this approach produces reasonable results for individual frames, video rendering usually suffers from temporal depth inconsistencies. Thus, we perform successive optimization to improve temporal consistency by fine-tuning our depth decoder for both temporal and spatial smoothness. Using a consumer-grade ODS camera, we evaluate our approach on a number of real-world scene recordings and demonstrate the versatility and robustness of the proposed pipeline.

show abstract

“…There are only few works [21]- [23] that estimate depth from pairs of views. The studies [21], [22] (and their prior analyses) estimate the relative pose between the cameras using traditional A-KAZE features [24] and the eight-point algorithm (8-PA) [25]; derotate and estimate dense features from the views; and refine the pose using an iterative nonlinear approach.…”

Section: Related Workmentioning

confidence: 99%

“…The studies [21], [22] (and their prior analyses) estimate the relative pose between the cameras using traditional A-KAZE features [24] and the eight-point algorithm (8-PA) [25]; derotate and estimate dense features from the views; and refine the pose using an iterative nonlinear approach. Lai and colleagues [23] present an encoderdecoder model that deals with stereo-rectfied image pairs with a small, fixed baseline. Their convolutional neural network, although not adapted to the spherical distortions, encourages the depth estimates from left and right boundaries to connect.…”

Section: Related Workmentioning

confidence: 99%

Dense 3D Indoor Scene Reconstruction from Spherical Images

Silveira¹,

Jung

2020

Anais Estendidos Da Conference on Graphics, Patterns and Images (SIBRAPI Estendido 2020)

View full text Add to dashboard Cite

Techniques for 3D reconstruction of scenes based on images are popular and support a number of secondary applications. Traditional approaches require several captures for covering whole environments due to the narrow field of view (FoV) of the pinhole-based/perspective cameras. This paper summarizes the main contributions of the homonym Ph.D. Thesis, which addresses the 3D scene reconstruction problem by considering omnidirectional (spherical or 360◦ ) cameras that present a 360◦ × 180◦ FoV. Although spherical imagery have the benefit of the full-FoV, they are also challenging due to the inherent distortions involved in the capture and representation of such images, which might compromise the use of many wellestablished algorithms for image processing and computer vision. The referred Ph.D. Thesis introduces novel methodologies for estimating dense depth maps from two or more uncalibrated and temporally unordered 360◦ images. It also presents a framework for inferring depth from a single spherical image. We validate our approaches using both synthetic data and computer-generated imagery, showing competitive results concerning other state-ofthe-art methods.

show abstract

Real-Time Panoramic Depth Maps from Omni-directional Stereo Images for 6 DoF Videos in Virtual Reality

Cited by 34 publications

References 24 publications

Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision

Panoramic Stereo Imaging of a Bionic Compound-Eye Based on Binocular Vision

Temporal Consistent Motion Parallax for Omnidirectional Stereo Panorama Video

Dense 3D Indoor Scene Reconstruction from Spherical Images

Contact Info

Product

Resources

About