Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Palazzi, Andrea; Bergamini, Luca; Calderara, Simone; Cucchiara, Rita

doi:10.1109/tpami.2020.3030701

Cited by 7 publications

(5 citation statements)

References 63 publications

(108 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address this challenge, Yin et al presented an unpaired view translation framework that used cVAE‐GAN to decompose the features of source views and control the generation of target views through view condition vectors 29 . Furthermore, Palazzi et al proposed a self‐supervised and semiparametric method (a fusion of an entirely learning‐based generative network and a not learned priori geometric knowledge component) that can generate novel views of a vehicle from a single monocular image 30 . The sparse‐view problem of MPI can refer to the solution in novel view synthesis.…”

Section: Related Workmentioning

confidence: 99%

“…29 Furthermore, Palazzi et al proposed a self -supervised and semiparametric method (a fusion of an entirely learning-based generative network and a not learned priori geometric knowledge component) that can generate novel views of a vehicle from a single monocular image. 30 The sparse-view problem of MPI can refer to the solution in novel view synthesis. Therefore, we proposed an PGNet that can generate novel projections to improve the 3D imaging temporal resolution of projection MPI.…”

Section: Novel View Synthesismentioning

confidence: 99%

See 1 more Smart Citation

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Gao

et al. 2022

Medical Physics

View full text Add to dashboard Cite

Background: Magnetic particle imaging (MPI) is a novel tomographic imaging modality that scans the distribution of superparamagnetic iron oxide nanoparticles. However, it is time-consuming to scan multiview two-dimensional (2D) projections for three-dimensional (3D) reconstruction in projection MPI, such as computed tomography (CT). An intuitive idea is to use the sparse-view projections for reconstruction to improve the temporal resolution. Tremendous progress has been made toward addressing the sparse-view problem in CT, because of the availability of large data sets. For the novel tomography of MPI, to the best of our knowledge, studies on the sparse-view problem have not yet been reported. Purpose: The acquisition of multiview projections for 3D MPI imaging is time-consuming. Our goal is to only acquire sparse-view projections for reconstruction to improve the 3D imaging temporal resolution of projection MPI. Methods: We propose to address the sparse-view problem in projection MPI by generating novel projections. The data set we constructed consists of three parts: simulation data set (including 3000 3D data), four phantoms data, and an in vivo mouse data. The simulation data set is used to train and validate the network, and the phantoms and in vivo mouse data are used to test the network. When the number of novel generated projections meets the requirements of filtered back projection, the streaking artifacts will be absent from MPI tomographic imaging. Specifically, we propose a projection generative network (PGNet), that combines an attention mechanism, adversarial training strategy, and a fusion loss function and can generate novel projections based on sparse-view real projections. To the best of our knowledge, we are the first to propose a deep learning method to attempt to overcome the sparse-view problem in projection MPI. Results: We compare our method with several sparse-view methods on phantoms and in vivo mouse data and validate the advantages and effectiveness of 2354

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Novel View Synthesismentioning

confidence: 99%

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Gao

et al. 2022

Medical Physics

View full text Add to dashboard Cite

show abstract

“…Pascal3D+. The Pascal3D+ dataset [54] contains images of 12 object classes, from both PASCAL VOC [5,10] and ImageNet [4], associated with 3D category-level models and coarse viewpoints [44,35,41,42]. Manuallyannotated foreground masks are available for the PAS-CAL VOC subset, while an off-the-shelf segmentation algorithm [11] is used for the other subset, as done in previous works [16,7,47].…”

Section: Datasets and Experimental Settingmentioning

confidence: 99%

Multi-Category Mesh Reconstruction From Image Collections

Simoni¹,

Pini²,

Vezzani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, learning frameworks have shown the capability of inferring the accurate shape, pose, and texture of an object from a single RGB image. However, current methods are trained on image collections of a single category in order to exploit specific priors, and they often make use of category-specific 3D templates. In this paper, we present an alternative approach that infers the textured mesh of objects combining a series of deformable 3D models and a set of instance-specific deformation, pose, and texture. Differently from previous works, our method is trained with images of multiple object categories using only foreground masks and rough camera poses as supervision. Without specific 3D templates, the framework learns category-level models which are deformed to recover the 3D shape of the depicted object. The instance-specific deformations are predicted independently for each vertex of the learned 3D mesh, enabling the dynamic subdivision of the mesh during the training process. Experiments show that the proposed framework can distinguish between different object categories and learn category-specific shape priors in an unsupervised manner. Predicted shapes are smooth and can leverage from multiple steps of subdivision during the training process, obtaining comparable or state-of-the-art results on two public datasets. Models and code are publicly released 1 .

show abstract

“…In [13], the model does not require 3D supervision, but the camera pose is needed to predict a dense flow field. Recently, Palazzi et al [34] generate novel views of objects in a semi-parametric setting: relying on both 3D CAD models and an image completion network. These previous methods can be utilized to solve vehicle view synthesis in a controlled 3D environment and require underlying 3D models or camera viewpoints.…”

Section: B Novel View Synthesismentioning

confidence: 99%

Pose-Based View Synthesis for Vehicles: A Perspective Aware Method

Sheng

Zhang

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

In this paper, we focus on the problem of novel view synthesis for vehicles. Some previous works solve the problem of novel view synthesis in a controlled 3D environment by exploiting additional 3D details (i.e., camera viewpoints and underlying 3D models). However, in real scenarios, the 3D details are difficult to obtain. In this case, we find that introducing vehicle pose to represent the views of vehicles is an alternative paradigm to solve the lack of 3D details. In novel view synthesis, preserving local details is one of the most challenging problems. To address this problem, we propose a perspective-aware generative model (PAGM). We are motivated by the prior that vehicles are made of quadrilateral planes. Preserving these rigid planes during image generation ensures that image details are kept. To this end, a classic image transformation method is leveraged, i.e., perspective transformation. In our GAN-based system, the perspective transformation is applied to the encoder feature maps, and the resulting maps are regarded as new conditions for the decoder. This strategy preserves the quadrilateral planes all the way through the network, thus shuttling the texture details from the input image to the generated image. In the experiments, we show that PAGM can generate high-quality vehicle images with fine details. Quantitatively, our method is superior to several competing approaches employing either GAN or the perspective transformation.

show abstract

Warp and Learn: Novel Views Generation for Vehicles and Other Objects

Cited by 7 publications

References 63 publications

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Multi-Category Mesh Reconstruction From Image Collections

Pose-Based View Synthesis for Vehicles: A Perspective Aware Method

Contact Info

Product

Resources

About