Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence

Sun, Shibo; Huh, Minyoung; Liao, Yuan-Hong; Zhang, Ning; Lim, Joseph J.

doi:10.1007/978-3-030-01219-9_10

Cited by 104 publications

(85 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though trained with 4 input images, we demonstrate that our networks can infer high-quality target images using fewer input images at test time. Using the experimental protocol of Sun et al 2018 [32], which uses up to 4 input images to infer a target image, we report quantitative results for our approach and others that can use multiple input images [32,33,45], as well as for an approach accepting single inputs [25].…”

Section: Novel View Synthesismentioning

confidence: 99%

“…This demonstrates that our method is able to generalize to intermediate poses not seen during training. In contrast, for their ShapeNet evaluations, [32] uses one-hot vectors indicating the discrete azimuth and elevation intervals at which the source images were rendered, and the specified pose for the target image. It is thus unclear how or whether their method would be able to generalize to intermediate poses not used for training.…”

Section: A411 Shapenet Chairs and Carsmentioning

confidence: 99%

“…1 viewTatarchenko et al 2015[33] .139 .875 .223 .882 Zhou et al 2016[45] .148 .877 .229 .871 Park et al 2017[25] .119 .913 .202 .889 Sun et al 2018[32] .098 .923 .181 .895 Ours .091 .927 .178 .895 2 views Tatarchenko et al 2015 [33] .124 .883 .209 .890 Zhou et al 2016 [45] .107 .901 .207 .881 Sun et al 2018 [32] .078 .935 .141 .911 Ours .072 .939 .136 .928 3 views Tatarchenko et al 2015 [33] .116 .887 .197 .898 Zhou et al 2016 [45] .089 .915 .188 .887 Sun et al 2018 [32] .068 .941 .122 .919 Ours .063 .943 .116 .936 4 views Tatarchenko et al 2015 [33] .112 .890 .192 .900 Zhou et al 2016 [45] .081 .924 .165 .891 Sun et al 2018 [32] .062 .946 .111 .925 Ours .059 .946 .107 .939…”

mentioning

confidence: 99%

See 2 more Smart Citations

Transformable Bottleneck Networks

Olszewski

Tulyakov

Woodford

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We propose a novel approach to performing fine-grained 3D manipulation of image content via a convolutional neural network, which we call the Transformable Bottleneck Network (TBN). It applies given spatial transformations directly to a volumetric bottleneck within our encoderbottleneck-decoder architecture. Multi-view supervision encourages the network to learn to spatially disentangle the feature space within the bottleneck. The resulting spatial structure can be manipulated with arbitrary spatial transformations. We demonstrate the efficacy of TBNs for novel view synthesis, achieving state-of-the-art results on a challenging benchmark. We demonstrate that the bottlenecks produced by networks trained for this task contain meaningful spatial structure that allows us to intuitively perform a variety of image manipulations in 3D, well beyond the rigid transformations seen during training. These manipulations include non-uniform scaling, non-rigid warping, and combining content from different images. Finally, we extract explicit 3D structure from the bottleneck, performing impressive 3D reconstruction from a single input image. 1

show abstract

Section: Novel View Synthesismentioning

confidence: 99%

Section: A411 Shapenet Chairs and Carsmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Transformable Bottleneck Networks

Olszewski

Tulyakov

Woodford

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…In a similar setting Tatarchenko et al [58] predicted both object appearance and depth map from different viewpoints. Successive works [76], [42] trained a network to learn a symmetry-aware appearance flow, re-casting the remaining synthesis as a task of image completion; [56] extends this approach to the case in which N > 1 input viewpoints are available. However, all these works [69], [58], [76], [42], [56] assume the target view to be known at training time.…”

Section: Related Workmentioning

confidence: 99%

“…Successive works [76], [42] trained a network to learn a symmetry-aware appearance flow, re-casting the remaining synthesis as a task of image completion; [56] extends this approach to the case in which N > 1 input viewpoints are available. However, all these works [69], [58], [76], [42], [56] assume the target view to be known at training time. As this is not usually the case in the real-world, these approaches limit themselves to train solely on synthetic data and exhibit limited generalization in a real-world scenario.…”

Section: Related Workmentioning

confidence: 99%

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Palazzi

Bergamini

Calderara

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

In this work we introduce a new self-supervised, semi-parametric approach for synthesizing novel views of a vehicle starting from a single monocular image. Differently from parametric (i.e. entirely learning-based) methods, we show how a-priori geometric knowledge about the object and the 3D world can be successfully integrated into a deep learning based image generation framework. As this geometric component is not learnt, we call our approach semi-parametric. In particular, we exploit man-made object symmetry and piece-wise planarity to integrate rich a-priori visual information into the novel viewpoint synthesis process. An Image Completion Network (ICN) is then trained to generate a realistic image starting from this geometric guidance. This careful blend between parametric and non-parametric components allows us to i) operate in a real-world scenario, ii) preserve high-frequency visual information such as textures, iii) handle truly arbitrary 3D roto-translations of the input and iv) perform shape transfer to completely different 3D models. Eventually, we show that our approach can be easily complemented with synthetic data and extended to other rigid objects with completely different topology, even in presence of concave structures and holes (e.g. chairs). A comprehensive experimental analysis against state-of-the-art competitors shows the efficacy of our method both from a quantitative and a perceptive point of view.

show abstract

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Gao

et al. 2022

Medical Physics

View full text Add to dashboard Cite

Background: Magnetic particle imaging (MPI) is a novel tomographic imaging modality that scans the distribution of superparamagnetic iron oxide nanoparticles. However, it is time-consuming to scan multiview two-dimensional (2D) projections for three-dimensional (3D) reconstruction in projection MPI, such as computed tomography (CT). An intuitive idea is to use the sparse-view projections for reconstruction to improve the temporal resolution. Tremendous progress has been made toward addressing the sparse-view problem in CT, because of the availability of large data sets. For the novel tomography of MPI, to the best of our knowledge, studies on the sparse-view problem have not yet been reported. Purpose: The acquisition of multiview projections for 3D MPI imaging is time-consuming. Our goal is to only acquire sparse-view projections for reconstruction to improve the 3D imaging temporal resolution of projection MPI. Methods: We propose to address the sparse-view problem in projection MPI by generating novel projections. The data set we constructed consists of three parts: simulation data set (including 3000 3D data), four phantoms data, and an in vivo mouse data. The simulation data set is used to train and validate the network, and the phantoms and in vivo mouse data are used to test the network. When the number of novel generated projections meets the requirements of filtered back projection, the streaking artifacts will be absent from MPI tomographic imaging. Specifically, we propose a projection generative network (PGNet), that combines an attention mechanism, adversarial training strategy, and a fusion loss function and can generate novel projections based on sparse-view real projections. To the best of our knowledge, we are the first to propose a deep learning method to attempt to overcome the sparse-view problem in projection MPI. Results: We compare our method with several sparse-view methods on phantoms and in vivo mouse data and validate the advantages and effectiveness of 2354

show abstract

Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence

Cited by 104 publications

References 33 publications

Transformable Bottleneck Networks

Transformable Bottleneck Networks

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Contact Info

Product

Resources

About