Deep Stereo: Learning to Predict New Views from the World's Imagery

Abstract

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision [20,29], but their use in graphics problems has been limited ([21, 7] are notable recent exceptions). In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches which consist of multiple complex stages of processing, each of which require careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. To verify our method we show that it can convincingly reproduce known test views from nearby imagery. Additionally we show images rendered from novel viewpoints. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.

View Synthesis by Appearance Flow

“…view extrapolation); 2) Unlike [27], who have a fixed number of input views, our multiview network is more flexible at both training and test time as it could take in an arbitrary number of input views for joint prediction, which is particularly beneficial when the number of input views varies at test time.…”

“…Buehler et al [26] presented a unifying framework for these image-based rendering techniques. The recent DeepStereo work by Flynn et al [27] is a learning-based extension that performs compositing through learned geometric reasoning using a CNN, and can generate intermediate views of a scene by interpolating from a set of surrounding views. While these methods yield high-quality novel views, they do so by composting the corresponding input image rays for each output pixel and can therefore only generate already seen content, (e.g.…”

“…While the general idea of learning hypothesis selection for view synthesis has been recently explored in [27], there are a few key differences between our framework and [27] method from synthesizing pixels that are invisible in the input views (i.e. view extrapolation); 2) Unlike [27], who have a fixed number of input views, our multiview network is more flexible at both training and test time as it could take in an arbitrary number of input views for joint prediction, which is particularly beneficial when the number of input views varies at test time.…”

How to Train a CAT: Learning Canonical Appearance Transformations for Direct Visual Localization Under Illumination Change

“…In particular, our work bears resemblances to [23], which generates such imagery using a convolutional neural network (CNN). In contrast to [23], our goal is not to learn an image synthesis pipeline to be queried at arbitrary poses, but rather to learn a correction to the appearance of a single image taken from a fixed pose.…”

“…Our work is related to the field of image-based rendering (e.g., [22], [23]), which aims to synthesize new views of a scene by blending existing images. In particular, our work bears resemblances to [23], which generates such imagery using a convolutional neural network (CNN).…”