Accelerating 3D Deep Learning with PyTorch3D

Abstract

Deep learning has significantly improved 2D image recognition. Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition. However despite growing interest, 3D deep learning remains relatively underexplored. We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable. We address these challenges by introducing PyTorch3D, a library of modular, efficient, and differentiable operators for 3D deep learning. It includes a fast, modular differentiable renderer for meshes and point clouds, enabling analysis-by-synthesis approaches. Compared with other differentiable renderers, PyTorch3D is more modular and efficient, allowing users to more easily extend it while also gracefully scaling to large meshes and images. We compare the PyTorch3D operators and renderer with other implementations and demonstrate significant speed and memory improvements. We also use PyTorch3D to improve the state-of-the-art for unsupervised 3D mesh and point cloud prediction from 2D images on ShapeNet. PyTorch3D is open-source and we hope it will help accelerate research in 3D deep learning. * equal contribution Preprint. Under review.

GeoFill: Reference-Based Image Inpainting of Scenes with Complex Geometry

“…Note that we also normalize the mesh to the unit size before rendering. We use Py-Torch3D [36] as our renderer. The textured mesh densely fills in pixels and removes hidden surfaces.…”

SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators

“…Because the rendering operation is normally discrete, it does not provide usable error gradients for optimization. A variety of mesh [10,33,29,24,31,8,43], point-cloud, and implicit [32,40,39,48] based differentiable renderers have been proposed. [24] developed an approximation of gradient for rasterization.…”

Recognizing Scenes from Novel Viewpoints

“…Modern techniques learn disparity from image pairs [30], estimate correspondences with contrastive learning [55], perform multi-view stereopsis via differentiable ray projection [28] or learn to reconstruct scenes while optimizing for cameras [26,48]. Differentiable rendering [10,29,36,38,41,47,50] allows gradients to flow to 3D scenes via 2D re-projections. [10,29,38,50] reconstruct single objects from a single view via rendering from 2 or more views during training.…”

“…In ad-dition to numerous semantic-specific details, recognition in novel viewpoints via direct appearance synthesis is suboptimal: one may be sure of the presence of a rug behind a couch, but unsure of its particular color. Similarly, there have been advances in learning to infer 3D properties of scenes from image cues [20,46,63], or with differentiable rendering [10,29,38,50] and other methods for bypassing the need for direct 3D supervision [27,33,34,68]. However, these approaches do not connect to complex scene semantics; they primarily focus on single objects or small, less diverse 3D annotated datasets.…”