Deep view synthesis from sparse photometric images

Abstract

The goal of light transport acquisition is to take images from a sparse set of lighting and viewing directions, and combine them to enable arbitrary relighting with changing view. While relighting from sparse images has received significant attention, there has been relatively less progress on view synthesis from a sparse set of "photometric" images---images captured under controlled conditions, lit by a single directional source; we use a spherical gantry to position the camera on a sphere surrounding the object. In this paper, we synthesize novel viewpoints across a wide range of viewing directions (covering a 60° cone) from a sparse set of just six viewing directions. While our approach relates to previous view synthesis and image-based rendering techniques, those methods are usually restricted to much smaller baselines, and are captured under environment illumination. At our baselines, input images have few correspondences and large occlusions; however we benefit from structured photometric images. Our method is based on a deep convolutional network trained to directly synthesize new views from the six input views. This network combines 3D convolutions on a plane sweep volume with a novel per-view per-depth plane attention map prediction network to effectively aggregate multi-view appearance. We train our network with a large-scale synthetic dataset of 1000 scenes with complex geometry and material properties. In practice, it is able to synthesize novel viewpoints for captured real data and reproduces complex appearance effects like occlusions, view-dependent specularities and hard shadows. Moreover, the method can also be combined with previous relighting techniques to enable changing both lighting and view, and applied to computer vision problems like multiview stereo from sparse image sets.

Free View Synthesis

“…Kalantari et al [16] use this idea for a light-field setup with a fixed number of cameras. Additional directional lighting extensions to these architectures enable synthesis of complex appearance effects [3,42].…”

Pose-Based View Synthesis for Vehicles: A Perspective Aware Method

“…The methods in novel view synthesis [12], [14], [22], [27]- [32] mainly fall into two categories: geometry-based approaches and appearance flow approaches. Geometry-based approaches benefit from geometric reasoning in solving the view synthesis problems.…”

Capture to Rendering Pipeline for Generating Dynamically Relightable Virtual Objects with Handheld RGB-D Cameras

“…However, these papers focused on reconstruction from a single image, which only has partial view of the target object, and is therefore not ideal for VR applications that allow objects to be viewed from any potential direction. Xu et al [44] recovered the texture of objects from multiple images. However, this approach required capture using a light stage, which is not allow for the possibility of portable scanning.…”