SynSin: End-to-End View Synthesis From a Single Image

Wiles, Olivia; Gkioxari, Georgia; Szeliski, Richard; Johnson, Justin

doi:10.1109/cvpr42600.2020.00749

Cited by 310 publications

(286 citation statements)

References 59 publications

Supporting

Mentioning

267

Contrasting

Order By: Relevance

“…A variety of supervisory signals have been proposed to learn such priors. Besides using 3D ground truth directly, authors have considered using videos [36], [45], [46], [47], [48], stereo pairs [38], [49] and multi-view images [50], [51], [52], [53], [54].…”

Section: Category-specific Reconstructionmentioning

confidence: 99%

Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild

Wu¹,

Rupprecht²,

Vedaldi³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We propose a method to learn 3D deformable object categories from raw single-view images, without external supervision. The method is based on an autoencoder that factors each input image into depth, albedo, viewpoint and illumination. In order to disentangle these components without supervision, we use the fact that many object categories have, at least approximately, a symmetric structure. We show that reasoning about illumination allows us to exploit the underlying object symmetry even if the appearance is not symmetric due to shading. Furthermore, we model objects that are probably, but not certainly, symmetric by predicting a symmetry probability map, learned end-to-end with the other components of the model. Our experiments show that this method can recover very accurately the 3D shape of human faces, cat faces and cars from single-view images, without any supervision or a prior shape model. On benchmarks, we demonstrate superior accuracy compared to another method that uses supervision at the level of 2D image correspondences.

show abstract

Section: Category-specific Reconstructionmentioning

confidence: 99%

Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild

Wu¹,

Rupprecht²,

Vedaldi³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…This research topic can be applied to various problems such as smoothing the transition between pairs of images, filling up missing regions, etc. Most methods make use of a 3D representation of the scene to synthesize novel views [7,8]. Inspired by [7,22], we adopt their 3D point cloud renderer in our architecture.…”

Section: Related Workmentioning

confidence: 99%

“…Most methods make use of a 3D representation of the scene to synthesize novel views [7,8]. Inspired by [7,22], we adopt their 3D point cloud renderer in our architecture. Our implementation is different in that we assume that camera poses are unknown and use a network for future camera pose estimation.…”

Section: Related Workmentioning

confidence: 99%

“…In the proposed architecture, we utilize adversarial losses to encourage realism. The use of adversarial training has been recently utilized in view synthesis [7]. Similarly, we adopt adversarial loss to enforce the realism criteria on the generated future frames.…”

Section: Related Workmentioning

confidence: 99%

“…Compared to the prior works in video prediction, our approach includes 3D awareness, which is based on recent advancements in novel view synthesis [7,8]. Given a set of previous frames, our method initially estimates camera poses for possible future frames.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Robust Multi-Frame Future Prediction By Leveraging View Synthesis

Sun

Lim

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

In this paper, we focus on the problem of video prediction, i.e., future frame prediction. Most state-of-the-art techniques focus on synthesizing a single future frame at each step. However, this leads to utilizing the model's own predicted frames when synthesizing multi-step prediction, resulting in gradual performance degradation due to accumulating errors in pixels. To alleviate this issue, we propose a model that can handle multi-step prediction. Additionally, we employ techniques to leverage from view synthesis for future frame prediction, where both problems are treated independently in the literature. Our proposed method employs multiview camera pose prediction and depth-prediction networks to project the last available frame to desired future frames via differentiable point cloud renderer. For the synthesis of moving objects, we utilize an additional refinement stage. In experiments, we show that the proposed framework outperforms state-of-theart methods in both KITTI and Cityscapes datasets.

show abstract