Depth synthesis and local warps for plausible image-based navigation

Abstract

Modern camera calibration and multiview stereo techniques enable users to smoothly navigate between different views of a scene captured using standard cameras. The underlying automatic 3D reconstruction methods work well for buildings and regular structures but often fail on vegetation, vehicles, and other complex geometry present in everyday urban scenes. Consequently, missing depth information makes Image-Based Rendering (IBR) for such scenes very challenging. Our goal is to provide plausible free-viewpoint navigation for such datasets. To do this, we introduce a new IBR algorithm that is robust to missing or unreliable geometry, providing plausible novel views even in regions quite far from the input camera positions. We first oversegment the input images, creating superpixels of homogeneous color content which often tends to preserve depth discontinuities. We then introduce a depth synthesis approach for poorly reconstructed regions based on a graph structure on the oversegmentation and appropriate traversal of the graph. The superpixels augmented with synthesized depth allow us to define a local shape-preserving warp which compensates for inaccurate depth. Our rendering algorithm blends the warped images, and generates plausible image-based novel views for our challenging target scenes. Our results demonstrate novel view synthesis in real time for multiple challenging scenes with significant depth complexity, providing a convincing immersive navigation experience.

Reminiscence Therapy using Image-Based Rendering in VR

“…Most IBR systems can be classified into two types -proxy based [5,15] and image warping approaches [7,6]. The former apply a view-dependent texture on the 3D model of the scene while the latter warp input images and blend the warped images to synthesize the final result.…”

“…Proxy-based approaches require an accurate 3D model which is essentially impossible to obtain using state of the art 3D reconstruction [17]. The very recent work on depth synthesis and warp-based rendering [6], does provide rendering quality which is acceptable for some level of VR display. The quality of the rendered images -given specific restrictions which we develop below -is definitely acceptable for the type of immersive VR we will use in this study, despite some residual visual artifacts.…”

“…The target hardware setup is a large screen with stereo display, and a high-end tracking system, which we describe next. We then describe the IBR approach of [6] we integrated and adapted to our system.…”

“…In contrast, image-based modeling and rendering [33], provides a much easier way to achieve this goal, since only a small set of photographs is required as input. It is however only very recently [6] that imagebased rendering (IBR) has achieved a sufficient level of quality and efficiency to be suitable for free-viewpoint navigation, and thus for VR. The high level of realism of [6] can be seen in Fig.…”

“…It is however only very recently [6] that imagebased rendering (IBR) has achieved a sufficient level of quality and efficiency to be suitable for free-viewpoint navigation, and thus for VR. The high level of realism of [6] can be seen in Fig. 1.…”