Accelerating 3D deep learning with PyTorch3D

Johnson, Justin; Ravi, Nikhila; Reizenstein, Jeremy; Novotný, David; Tulsiani, Shubham; Lassner, Christoph; Branson, Steve

doi:10.1145/3415263.3419160

Cited by 44 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the decoder part, we have four components, each of which is to be trained in this stage: (1) the trained shape basis of SFM, (2) the expression basis D exp from BFM2017 [14], (3) the albedo basis D albedo from Ref. [59], (4) the rendering layer takes the geometric, albedo, illumination, pose parameter, and camera parameter and renders 224×224 RGB images, which is based on Pytorch3d [60]. The illumination model is a spherical harmonic illumination model.…”

Section: Learning Frameworkmentioning

confidence: 99%

Sphere Face Model: A 3D morphable model with hypersphere manifold latent space using joint 2D/3D training

et al. 2023

View full text Add to dashboard Cite

Abstract3D morphable models (3DMMs) are generative models for face shape and appearance. Recent works impose face recognition constraints on 3DMM shape parameters so that the face shapes of the same person remain consistent. However, the shape parameters of traditional 3DMMs satisfy the multivariate Gaussian distribution. In contrast, the identity embeddings meet the hypersphere distribution, and this conflict makes it challenging for face reconstruction models to preserve the faithfulness and the shape consistency simultaneously. In other words, recognition loss and reconstruction loss can not decrease jointly due to their conflict distribution. To address this issue, we propose the Sphere Face Model (SFM), a novel 3DMM for monocular face reconstruction, preserving both shape fidelity and identity consistency. The core of our SFM is the basis matrix which can be used to reconstruct 3D face shapes, and the basic matrix is learned by adopting a two-stage training approach where 3D and 2D training data are used in the first and second stages, respectively. We design a novel loss to resolve the distribution mismatch, enforcing that the shape parameters have the hyperspherical distribution. Our model accepts 2D and 3D data for constructing the sphere face models. Extensive experiments show that SFM has high representation ability and clustering performance in its shape parameter space. Moreover, it produces high-fidelity face shapes consistently in challenging conditions in monocular face reconstruction. The code will be released at https://github.com/a686432/SIR

show abstract

Section: Learning Frameworkmentioning

confidence: 99%

Sphere Face Model: A 3D morphable model with hypersphere manifold latent space using joint 2D/3D training

et al. 2023

View full text Add to dashboard Cite

show abstract

“…During training, the 3D shape is rendered at 5 views, whose pitch and yaw angles are (0 , 0 • ), and the L2 distances of the rendered images between the output and the ground-truth 3D shapes are formulated as the visual-effect distance: where v is the view index, R(•, •) is the renderer whose input is 3D shape and texture, R v is the rotation matrix corresponding to each view, and T w is the all-white texture under orthogonal light. If we employ a differentiable renderer [54], the D psd can be directly regarded as a loss function. As shown in Fig.…”

Section: Plaster Sculpture Descriptormentioning

confidence: 99%

Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape

Zhu

Huang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. However, previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry, which is attributed to insufficient ground-truth 3D shapes, unreliable training strategies and limited representation power of 3DMM. To alleviate this issue, this paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person. Specifically, given a 2D image as the input, we virtually render the image in several calibrated views to normalize pose variations while preserving the original image geometry. A many-to-one hourglass network serves as the encode-decoder to fuse multiview features and generate vertex displacements as the fine-grained geometry. Besides, the neural network is trained by directly optimizing the visual effect, where two 3D shapes are compared by measuring the similarity between the multiview images rendered from the shapes. Finally, we propose to generate the ground-truth 3D shapes by registering RGB-D images followed by pose and shape augmentation, providing sufficient data for network training. Experiments on several challenging protocols demonstrate the superior reconstruction accuracy of our proposal on the face shape.

show abstract

“…The incomplete texture map is run through a generator network. The synthesized texture map is then wrapped to the face mesh and the mesh is rendered using a diferentiable renderer [22] from a randomly sampled viewpoint. The random sampling ensures that areas of the texture map which were occluded in the original input image are now visible on the rendered image.…”

Section: Uv Texture Completionmentioning

confidence: 99%

Deep Learning Augmented Realistic Avatars for Social VR Human Representation

Boon

Fermoselle

Haar

et al. 2022

ACM International Conference on Interactive Media Experiences

View full text Add to dashboard Cite

Virtual reality (VR) has created a new and rich medium for people to meet each other digitally. In VR, people can choose from a broad range of representations. In several cases, it is important to provide users with avatars that are a lifelike representation of themselves, to increase the user experience and efectiveness of communication. In this work, we propose a pipeline for generating a realistic and expressive avatar from a single reference image. The pipeline consists of a blendshape-based avatar combined with two deep learning improvements. The frst improvement module runs ofine and improves the texture map of the base avatar. The second module runs inference in real-time at the rendering stage and performs a style transfer to the avatar's eyes. The deep learning modules efectively improve the visual representation of the avatar and show how AI techniques can be integrated with traditional animation methods to generate realistic human avatars for social VR.

show abstract

Accelerating 3D deep learning with PyTorch3D

Cited by 44 publications

References 0 publications

Sphere Face Model: A 3D morphable model with hypersphere manifold latent space using joint 2D/3D training

Sphere Face Model: A 3D morphable model with hypersphere manifold latent space using joint 2D/3D training

Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape

Deep Learning Augmented Realistic Avatars for Social VR Human Representation

Contact Info

Product

Resources

About