Cycle-Consistent Generative Rendering for 2D-3D Modality Translation

Abstract

Neural Radiance Fields (NeRFs) have proven to be powerful 3D representations, capable of high quality novel view synthesis of complex scenes. While NeRFs have been applied to graphics, vision, and robotics, problems with slow rendering speed and characteristic visual artifacts prevent adoption in many use cases. In this work, we investigate combining an autoencoder (AE) with a NeRF, in which latent features (instead of colours) are rendered and then convolutionally decoded. The resulting latent-space NeRF can produce novel views with higher quality than standard colour-space NeRFs, as the AE can correct certain visual artifacts, while rendering over three times faster. Our work is orthogonal to other techniques for improving NeRF efficiency. Further, we can control the tradeoff between efficiency and image quality by shrinking the AE architecture, achieving over 13 times faster rendering with only a small drop in performance. We hope that our approach can form the basis of an efficient, yet high-fidelity, 3D scene representation for downstream tasks, especially when retaining differentiability is useful, as in many robotics scenarios requiring continual learning.

Representing 3D Shapes with Probabilistic Directed Distance Fields

“…, where we choose ς ∈ {−1, 1} such that n T v < 0 (so that n always points back to the query oriented point) 1 . In this sense, n(p, v) is the visible surface normal on S, as seen from (p, v).…”

“…Finally, we apply CPDDFs to 3D-aware generative modelling, using 2D-3D unpaired data (see, e.g., [1,26,45,87]). This takes advantage of 3D model data, yet avoids requiring paired data.…”

“…9 displays results on ShapeNet cars [4,6], including disentanglement of shape, viewpoint, and appearance (see also, e.g., [18,26,62]). While this underperforms a 2D image GAN (15 versus 27 FID [19,49]), it still outperforms samples from image VAEs or GAN-based textured low-poly 3D mesh renders (>100 FID [1]) in image quality. See Appendix F for details.…”

“…where L A is the adversarial loss for the generator and the last two terms enforce consistency (see also [1,26,45]). We fit to the dataset of ShapeNet renders from Choy et al [6].…”

“…Similarly, implicit geometric fields, such as occupancy fields [43] and signed distance fields (SDFs) [51], have recently been used in conjunction with differentiable renderering as well [24,38,48]. Other works have learned textured mesh inference and/or generation via rendering-based approaches (e.g., [1,12,18,53,71]).…”