Transformable Bottleneck Networks

Olszewski, Kyle; Tulyakov, Sergey; Woodford, Oliver J.; Li, Hao; Luo, Linjie

doi:10.1109/iccv.2019.00774

Cited by 71 publications

(72 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After our initial submission, several new related works have been published by the scientific community. We include here a non exhaustive list of them; [36], [68], [55], [37], [38], [7].…”

Section: Related Workmentioning

confidence: 99%

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Palazzi

Bergamini

Calderara

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

In this work we introduce a new self-supervised, semi-parametric approach for synthesizing novel views of a vehicle starting from a single monocular image. Differently from parametric (i.e. entirely learning-based) methods, we show how a-priori geometric knowledge about the object and the 3D world can be successfully integrated into a deep learning based image generation framework. As this geometric component is not learnt, we call our approach semi-parametric. In particular, we exploit man-made object symmetry and piece-wise planarity to integrate rich a-priori visual information into the novel viewpoint synthesis process. An Image Completion Network (ICN) is then trained to generate a realistic image starting from this geometric guidance. This careful blend between parametric and non-parametric components allows us to i) operate in a real-world scenario, ii) preserve high-frequency visual information such as textures, iii) handle truly arbitrary 3D roto-translations of the input and iv) perform shape transfer to completely different 3D models. Eventually, we show that our approach can be easily complemented with synthetic data and extended to other rigid objects with completely different topology, even in presence of concave structures and holes (e.g. chairs). A comprehensive experimental analysis against state-of-the-art competitors shows the efficacy of our method both from a quantitative and a perceptive point of view.

show abstract

“…After our initial submission, several new related works have been published by the scientific community. We include here a non exhaustive list of them; [36], [68], [55], [37], [38], [7].…”

Section: Related Workmentioning

confidence: 99%

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Palazzi

Bergamini

Calderara

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Recent novel view synthesis methods approach the single-image setting using deep learning [Tatarchenko et al 2015;Zhou et al 2016]. Synthesizing novel views from a single image is inherently challenging and existing methods are often only applicable to specific scene types [Habtegebrial et al 2018;Nguyen-Phuoc et al 2019], 3D object models [Olszewski et al 2019;Park et al 2017;Rematas et al 2017;Yan et al 2016;Yang et al 2015], or domainspecific light field imagery [Srinivasan et al 2017]. Most relevant to our work are methods that estimate the scene geometry of the input image via depth [Cun et al 2019;], normal maps , or layered depth [Tulsiani et al 2018].…”

Section: Learning-based View Synthesis From a Single Imagementioning

confidence: 99%

3D Ken Burns effect from a single image

et al. 2019

View full text Add to dashboard Cite

from to Fig. 1. 3D Ken Burns effect from a single image. Given a single input image and optional user annotations in form of two cropping windows, our framework animates the input image while adding parallax to synthesize a 3D Ken Burns effect. Our method works well for a wide variety of content, including portrait (top) and landscape (bottom) photos. Please refer to our supplementary video demo for more examples. Please note that this figure, as well as many other figures in this paper, contain video clips. Should these videos not already be playing then please consider viewing this paper using Adobe Reader.The Ken Burns effect allows animating still images with a virtual camera scan and zoom. Adding parallax, which results in the 3D Ken Burns effect, enables significantly more compelling results. Creating such effects manually is timeconsuming and demands sophisticated editing skills. Existing automatic methods, however, require multiple input images from varying viewpoints. In this paper, we introduce a framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework first leverages a depth prediction pipeline, which estimates scene depth that is suitable for view synthesis tasks. To address the limitations of existing depth estimation methods such as geometric distortions, semantic distortions, and inaccurate depth boundaries, we develop a semantic-aware neural network for depth prediction, couple its estimate with a segmentation-based depth adjustment process, and employ a refinement neural network that facilitates accurate depth predictions at object boundaries. According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions. To address disocclusions while maintaining geometrically and temporally coherent synthesis results, we utilize contextaware color-and depth-inpainting to fill in the missing information in the extreme views of the camera path, thus extending the scene geometry of the point cloud. Experiments with a wide variety of image content show that our method enables realistic synthesis results. Our study demonstrates that our system allows users to achieve better results while requiring little effort compared to existing solutions for the 3D Ken Burns effect creation.

show abstract

“…Recent work leverages deep neural networks to learn a monocular image-toimage mapping between source and target view from data [29,52,8,62,40,51,59]. One line of work [29,52,8,39] directly generates image pixels. Given the difficulty of the task, direct image-to-image translation approaches struggle with preservation of local details and often produce blurry images.…”

Section: Related Workmentioning

confidence: 99%

Monocular Neural Image Based Rendering With Continuous View Control

Song

Chen

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Interactive novel view synthesis: Given a single source view our approach can generate a continuous sequence of geometrically accurate novel views under fine-grained control. Top: Given a single street-view like input, a user may specify a continuous camera trajectory and our system generates the corresponding views in real-time. Bottom: An unseen hires internet image is used to synthesize novel views, while the camera is controlled interactively. Please refer to our project homepage † .

show abstract

Transformable Bottleneck Networks

Cited by 71 publications

References 32 publications

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

3D Ken Burns effect from a single image

Monocular Neural Image Based Rendering With Continuous View Control

Contact Info

Product

Resources

About