Peeking behind objects: Layered depth prediction from a single image

Abstract

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.

Object-Driven Multi-Layer Scene Decomposition From a Single Image

“…Unlike Dhamo et al [6], we employ a mesh-based rendering approach. The advantage is that the 3D mesh captures all the available information in the scene, while an image-based approach [6] only captures information which is present in the set of consecutive image frames to be warped. For every frame, we render the visible instances separately, similarly to [7].…”

“…For example, we do not want the object from another room (behind the enclosing wall of the currently visible room) to form part in the compositional layers of that view. The advantage of the proposed semantic-aware rendering with respect to [41,6] is that it enables learning of class specific features, which might turn helpful in regressing plausible objects in the novel LDI layers.…”

“…Recent works generate such a data representation either from multi-view [14], or stereo [50] input, with some variations in the representation. More adventurous approaches [6,41] aim to learn an LDI from a single color image. The motivation is providing a method that does not rely on the availability of appropriate photo pairs/sets, so that consumers can reconstruct a 3D photo out of any image, casually at hand.…”

“…The motivation is providing a method that does not rely on the availability of appropriate photo pairs/sets, so that consumers can reconstruct a 3D photo out of any image, casually at hand. Both Dhamo et al [6] and Tulsiani et al [41] report results with LDIs having two layers only (background and foreground).…”

“…First, we propose a flexible extension for a multiple layer representation, where unlike [41], the number of layers is not pre-defined in the form of CNN architecture branches. Second, we leverage predictions of semantic identities to perform object-oriented completions, which are not considered in the generation procedure of previous methods [6,41]. Third, we propose a re-composition loss that is specific to our task.…”