Depth Image Based View Synthesis With Multiple Reference Views for Virtual Reality

Abstract

This paper presents a method for view synthesis from multiple views and their depth maps for free navigation in Virtual Reality with six degrees of freedom (6DoF) and 360 video (3DoF+), including synthesizing views corresponding to stepping in or out of the scene. Such scenarios should support large baseline view synthesis, typically going beyond the view synthesis involved in light field displays [1]. Our method allows to input an unlimited number of reference views, instead of the usual left and right reference views. Increasing the number of reference views overcomes problems such as occlusions, tangential surfaces to the cameras axis and artifacts in low quality depth maps. We outperform MPEG's reference software, VSRS [2], with a gain of up to 2.5 dB in PSNR when using four reference views.Index Terms -View synthesis, depth image based rendering, free navigation

Light Field Rendering for non-Lambertian Objects

“…IBR techniques overcome this problem by warping the image pixels to the new images in screen space. An specific IBR technique, Depth Image-Based Rendering (DIBR) [10,8,33,4], warps the input images depending on the motion parallax which is inversely proportional to the depth of the scene. The Immersive sub-community of the Moving Picture Experts Group (MPEG-I, developing standards for immersive video compression) divides DIBR in two steps after the cameras calibration: 1) depth estimation and 2) rendering.…”

Computer Generated Holography with Depth-based View Synthesis

“…The working principles of RVS are outlined in [5] (for its predecessor called SVS) and are very similar to those of [1]. The main differences are that (i) a disparity is given per pixel (a dense disparity map), (ii) the input camera views do not have to be parallel (the pixel shift is then replaced by a reprojection of an input view to the virtual view) and (iii) any triplet of adjacent pixels are implicitly connected through triangles in the OpenGL rendering process (which is the cause of the in Fig.…”

MPEG-I Depth Estimation Reference Software

“…Where E * (l) is the smallest error among the different search images for label l. It is computed using D * as the baseline between the reference image and the search image from which we selected E * (l), M R as the reliability map and C ij (l) as the matching costs found in (6).…”

“…Estimating high-quality depth remains a delicate process, but it is also crucial for creating synthesized views, where any depth imperfection may have a detrimental impact on the output quality [7,6]. So far, the Depth Estimation Reference Software (DERS) has been used (and reused from previous activities) for over 7 years and a lot of work has been invested, currently ending up in a quite powerful software: DERS8.0.…”