Deep View Morphing

Ji, Dinghuang; Junghyun, Kwon; McFarland, Max; Savarese, Silvio

doi:10.1109/cvpr.2017.750

Cited by 65 publications

(58 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Novel view synthesis is typically solved using image based rendering techniques [Kang et al 2006], with recent approaches allowing for high-quality view synthesis results [Chaurasia et al 2013[Chaurasia et al , 2011Hedman et al 2017;Hedman and Kopf 2018;Penner and Zhang 2017]. With the emergence of deep neural networks, learning-based techniques have become an increasingly popular tool for novel view synthesis [Flynn et al 2016;Ji et al 2017;Kalantari et al 2016;Meshry et al 2019;Mildenhall et al 2019;Sitzmann et al 2019;Thies et al , 2018Zhou et al 2018]. To enable high-quality synthesis results, existing methods typically require multiple input views [Kang et al 2006;Penner and Zhang 2017].…”

Section: Related Work 21 Novel View Synthesismentioning

confidence: 99%

3D Ken Burns effect from a single image

et al. 2019

View full text Add to dashboard Cite

from to Fig. 1. 3D Ken Burns effect from a single image. Given a single input image and optional user annotations in form of two cropping windows, our framework animates the input image while adding parallax to synthesize a 3D Ken Burns effect. Our method works well for a wide variety of content, including portrait (top) and landscape (bottom) photos. Please refer to our supplementary video demo for more examples. Please note that this figure, as well as many other figures in this paper, contain video clips. Should these videos not already be playing then please consider viewing this paper using Adobe Reader.The Ken Burns effect allows animating still images with a virtual camera scan and zoom. Adding parallax, which results in the 3D Ken Burns effect, enables significantly more compelling results. Creating such effects manually is timeconsuming and demands sophisticated editing skills. Existing automatic methods, however, require multiple input images from varying viewpoints. In this paper, we introduce a framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework first leverages a depth prediction pipeline, which estimates scene depth that is suitable for view synthesis tasks. To address the limitations of existing depth estimation methods such as geometric distortions, semantic distortions, and inaccurate depth boundaries, we develop a semantic-aware neural network for depth prediction, couple its estimate with a segmentation-based depth adjustment process, and employ a refinement neural network that facilitates accurate depth predictions at object boundaries. According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions. To address disocclusions while maintaining geometrically and temporally coherent synthesis results, we utilize contextaware color-and depth-inpainting to fill in the missing information in the extreme views of the camera path, thus extending the scene geometry of the point cloud. Experiments with a wide variety of image content show that our method enables realistic synthesis results. Our study demonstrates that our system allows users to achieve better results while requiring little effort compared to existing solutions for the 3D Ken Burns effect creation.

show abstract

Section: Related Work 21 Novel View Synthesismentioning

confidence: 99%

3D Ken Burns effect from a single image

et al. 2019

View full text Add to dashboard Cite

show abstract

“…With a huge amount of multiview images, 3D stereo algorithms [5] are applicable to reconstruct the 3D scene and then be utilized to synthesize novel views. Ji et al [11] proposed to synthesize middle view images by using two rectified view images. Yan et al [33] proposed a perspective transformer network to learn the projection transformation after reconstructing the 3D volume of the object.…”

Section: Related Workmentioning

confidence: 99%

“…View synthesis is a long-standing problem in computer vision [5,11,25,33,35], which facilitates many applications including surrounding perception and virtual reality. In modern autonomous driving solution, the limited viewpoint of on-car cameras restricts the system from reliably understanding the environment, acquiring accurate global view for better policy making and path planning.…”

Section: Introductionmentioning

confidence: 99%

Generative Adversarial Frontal View to Bird View Synthesis

Zhu

Yin²,

Shi³

et al. 2018

2018 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

Environment perception is an important task with great practical value and bird view is an essential part for creating panoramas of surrounding environment. Due to the large gap and severe deformation between the frontal view and bird view, generating a bird view image from a single frontal view is challenging. To tackle this problem, we propose the BridgeGAN, i.e., a novel generative model for bird view synthesis. First, an intermediate view, i.e., homography view, is introduced to bridge the large gap. Next, conditioned on the three views (frontal view, homography view and bird view) in our task, a multi-GAN based model is proposed to learn the challenging cross-view translation. Extensive experiments conducted on a synthetic dataset have demonstrated that the images generated by our model are much better than those generated by existing methods, with more consistent global appearance and sharper details. Ablation studies and discussions show its reliability and robustness in some challenging cases. Codes are available at

show abstract

“…More recent work [Ji et al 2017;] employ some notion of 3D geometry in the end-to-end process to deal with the 2D-3D object mapping. For instance, use an explicit flow that maps pixels from the input image to the output novel view.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, use an explicit flow that maps pixels from the input image to the output novel view. In Deep View Morphing [Ji et al 2017] two input images and an explicit rectification stage, that roughly aligns the inputs, are used to generate intermediate views. split the problem between visible pixels, i.e.…”

Section: Related Workmentioning

confidence: 99%

LookinGood

et al. 2018

View full text Add to dashboard Cite

Fig. 1. LookinGood leverages recent advances in real-time 3D performance capture and machine learning to re-render high quality novel viewpoints of a captured scene. A textured 3D reconstruction is first rendered to a novel viewpoint. Due to imperfections in geometry and low-resolution texture, the 2D rendered image contains artifacts and is low quality. Therefore we propose a deep learning technique that takes these images as input and generates more visually enhanced re-rendering. The system is specifically designed for VR and AR headsets, and accounts for consistency between two stereo views.Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and * Authors equally contributed to this work. our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 • capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.

show abstract

Deep View Morphing

Cited by 65 publications

References 36 publications

3D Ken Burns effect from a single image

3D Ken Burns effect from a single image

Generative Adversarial Frontal View to Bird View Synthesis

LookinGood

Contact Info

Product

Resources

About