RGBD-Net: Predicting color and depth images for novel views synthesis

Abstract

We address the problem of novel view synthesis from an unstructured set of reference images. A new method called RGBD-Net is proposed to predict the depth map and the color images at the target pose in a multi-scale manner. The reference views are warped to the target pose to obtain multi-scale plane sweep volumes, which are then passed to our first module, a hierarchical depth regression network which predicts the depth map of the novel view. Second, a depth-aware generator network refines the warped novel views and renders the final target image. These two networks can be trained with or without depth supervision. In experimental evaluation, RGBD-Net not only produces novel views with higher quality than the previous state-ofthe-art methods, but also the obtained depth maps enable reconstruction of more accurate 3D point clouds than the existing multi-view stereo methods. The results indicate that RGBD-Net generalizes well to previously unseen data.

FaDIV-Syn: Fast Depth-Independent View Synthesis using Soft Masks and Implicit Blending

“…A large variety of novel view synthesis approaches have been developed. For a broader review, we refer to Nguyen et al [21] or Tewari et al [37].…”

“…Image Based Rendering (IBR). In contrast to classical rendering of 3D scenes using textured geometry, IBR methods aim to render novel views by combining input images in the target pose [25,12,11,3,41,15,21,43,38,27,28]. To be able to project the input images correctly, IBR methods still require geometry, often in the form of depth maps, which are either available or estimated.…”

“…They use the corner cameras to predict depth in the target view, which is then used to warp the input views. Nguyen et al [21] introduce RGBD-Net, which first estimates depth using a multi-scale PSV, warps the input images into the target frame, performs explicit blending, and refines the warped image using a depth-aware network. Our method also works directly on the input images, but does not compute or require depth explicitly.…”

GeoNeRF: Generalizing NeRF with Geometry Priors

“…Concurrent to our work, the followings also introduce a generalizable NeRF: RGBD-Net [37] builds a cost volume for the target view instead of source views, NeuralMVS [44] proposes a coarse to fine approach to increase speed, and NeuRay [31] proposes a method to deal with occlusions.…”

Free-Viewpoint RGB-D Human Performance Capture and Rendering

“…Novel view synthesis of rigid objects or dynamic scenes has been a very active topic of research recently with impressive results across various tasks [43,48,65,68]. However, synthesizing novel views of humans in motion requires methods to handle dynamic scenes with various deformations which is a challenging task [66,72]; especially in those regions with fine details such as the face or the clothes [49,51,71].…”